Merge pull request #269222 from PatrickFarley/content-safety-updates

[ai svcs] Content safety March new features

Author: v-dirichards

articles/ai-services/content-safety/concepts/groundedness.md

@@ -0,0 +1,78 @@
+---
+title: "Groundedness detection in Azure AI Content Safety"
+titleSuffix: Azure AI services
+description: Learn about groundedness in large language model (LLM) responses, and how to detect outputs that deviate from source material.
+#services: cognitive-services
+author: PatrickFarley
+manager: nitinme
+ms.service: azure-ai-content-safety
+ms.topic: conceptual
+ms.date: 03/15/2024
+ms.author: pafarley
+---
+
+#  Groundedness detection
+
+The Groundedness detection API detects whether the text responses of large language models (LLMs) are grounded in the source materials provided by the users. Ungroundedness refers to instances where the LLMs produce information that is non-factual or inaccurate from what was present in the source materials.
+
+
+## Key terms
+
+- **Retrieval Augmented Generation (RAG)**: RAG is a technique for augmenting LLM knowledge with other data. LLMs can reason about wide-ranging topics, but their knowledge is limited to the public data that was available at the time they were trained. If you want to build AI applications that can reason about private data or data introduced after a model’s cutoff date, you need to provide the model with that specific information. The process of bringing the appropriate information and inserting it into the model prompt is known as Retrieval Augmented Generation (RAG). For more information, see [Retrieval-augmented generation (RAG)](https://python.langchain.com/docs/use_cases/question_answering/).
+
+- **Groundedness and Ungroundedness in LLMs**: This refers to the extent to which the model’s outputs are based on provided information or reflect reliable sources accurately. A grounded response adheres closely to the given information, avoiding speculation or fabrication. In groundedness measurements, source information is crucial and serves as the grounding source.
+
+## Groundedness detection features
+
+- **Domain Selection**: Users can choose an established domain to ensure more tailored detection that aligns with the specific needs of their field. Currently the available domains are `MEDICAL` and `GENERIC`.
+- **Task Specification**: This feature lets you select the task you're doing, such as QnA (question & answering) and Summarization, with adjustable settings according to the task type.
+- **Speed vs Interpretability**: There are two modes that trade off speed with result interpretability.
+   - Non-Reasoning mode: Offers fast detection capability; easy to embed into online applications.
+   - Reasoning mode: Offers detailed explanations for detected ungrounded segments; better for understanding and mitigation.
+
+##  Use cases
+
+Groundedness detection supports text-based Summarization and QnA tasks to ensure that the generated summaries or answers are accurate and reliable. Here are some examples of each use case:
+
+**Summarization tasks**:
+- Medical summarization: In the context of medical news articles, Groundedness detection can be used to ensure that the summary doesn't contain fabricated or misleading information, guaranteeing that readers obtain accurate and reliable medical information.
+- Academic paper summarization: When the model generates summaries of academic papers or research articles, the function can help ensure that the summarized content accurately represents the key findings and contributions without introducing false claims.
+
+**QnA tasks**:
+- Customer support chatbots: In customer support, the function can be used to validate the answers provided by AI chatbots, ensuring that customers receive accurate and trustworthy information when they ask questions about products or services.
+- Medical QnA: For medical QnA, the function helps verify the accuracy of medical answers and advice provided by AI systems to healthcare professionals and patients, reducing the risk of medical errors.
+- Educational QnA: In educational settings, the function can be applied to QnA tasks to confirm that answers to academic questions or test prep queries are factually accurate, supporting the learning process.
+
+## Limitations
+
+### Language availability
+
+Currently, the Groundedness detection API supports English language content. While our API doesn't restrict the submission of non-English content, we can't guarantee the same level of quality and accuracy in the analysis of other language content. We recommend that users submit content primarily in English to ensure the most reliable and accurate results from the API.
+
+### Text length limitations
+
+The maximum character limit for the grounding sources is 55,000 characters per API call, and for the text and query, it's 7,500 characters per API call. If your input (either text or grounding sources) exceeds these character limitations, you'll encounter an error.
+
+### Regions
+
+To use this API, you must create your Azure AI Content Safety resource in the supported regions. Currently, it's available in the following Azure regions:
+- East US 2
+- East US (only for non-reasoning)
+- West US
+- Sweden Central
+
+### TPS limitations
+
+| Pricing Tier | Requests per 10 seconds |
+| :----------- | :--------------------------- |
+| F0           | 10                           |
+| S0           | 10                           |
+
+If you need a higher rate, [contact us](mailto:[email protected]) to request it.
+
+## Next steps
+
+Follow the quickstart to get started using Azure AI Content Safety to detect groundedness.
+
+> [!div class="nextstepaction"]
+> [Groundedness detection quickstart](../quickstart-groundedness.md)

articles/ai-services/content-safety/concepts/jailbreak-detection.md

@@ -1,47 +1,90 @@
 ---
-title: "Jailbreak risk detection in Azure AI Content Safety"
+title: "Prompt Shields in Azure AI Content Safety"
 titleSuffix: Azure AI services
-description: Learn about jailbreak risk detection and the related flags that the Azure AI Content Safety service returns.
+description: Learn about User Prompt injection attacks and the Prompt Shields feature that helps prevent them.
 #services: cognitive-services
 author: PatrickFarley
 manager: nitinme
 ms.service: azure-ai-content-safety
 ms.custom: build-2023
 ms.topic: conceptual
-ms.date: 11/07/2023
+ms.date: 03/15/2024
 ms.author: pafarley
 ---
 
+# Prompt Shields
 
-# Jailbreak risk detection
+Generative AI models can pose risks of exploitation by malicious actors. To mitigate these risks, we integrate safety mechanisms to restrict the behavior of large language models (LLMs) within a safe operational scope. However, despite these safeguards, LLMs can still be vulnerable to adversarial inputs that bypass the integrated safety protocols.
 
+Prompt Shields is a unified API that analyzes LLM inputs and detects User Prompt attacks and Document attacks, which are two common types of adversarial inputs.
 
-Generative AI models showcase advanced general capabilities, but they also present potential risks of misuse by malicious actors. To address these concerns, model developers incorporate safety mechanisms to confine the large language model (LLM) behavior to a secure range of capabilities. Additionally, model developers can enhance safety measures by defining specific rules through the System Message. 
+### Prompt Shields for User Prompts
 
-Despite these precautions, models remain susceptible to adversarial inputs that can result in the LLM completely ignoring built-in safety instructions and the System Message. 
+Previously called **Jailbreak risk detection**, this shield targets User Prompt injection attacks, where users deliberately exploit system vulnerabilities to elicit unauthorized behavior from the LLM. This could lead to inappropriate content generation or violations of system-imposed restrictions.
 
-## What is a jailbreak attack?
+### Prompt Shields for Documents
 
-A jailbreak attack, also known as a User Prompt Injection Attack (UPIA), is an intentional attempt by a user to exploit the vulnerabilities of an LLM-powered system, bypass its safety mechanisms, and provoke restricted behaviors. These attacks can lead to the LLM generating inappropriate content or performing actions restricted by System Prompt or RLHF(Reinforcement Learning with Human Feedback).  
+This shield aims to safeguard against attacks that use information not directly supplied by the user or developer, such as external documents or images. Attackers might embed hidden instructions in these materials in order to gain unauthorized control over the LLM session.
 
-Most generative AI models are prompt-based: the user interacts with the model by entering a text prompt, to which the model responds with a completion.  
+## Types of input attacks
 
-Jailbreak attacks are User Prompts designed to provoke the Generative AI model into exhibiting behaviors it was trained to avoid or to break the rules set in the System Message. These attacks can vary from intricate role-play to subtle subversion of the safety objective. 
+The two types of input attacks that Prompt Shields detects are described in this table.
 
-## Types of jailbreak attacks
+| Type | Attacker | Entry point    | Method    | Objective/impact   | Resulting behavior  |
+|-------|----------|---------|---------|---------|---------|
+| User Prompt attacks | User     | User prompts      | Ignoring system prompts/RLHF training  | Altering intended LLM behavior         | Performing restricted actions against training |
+| Document attacks   | Third party | Third-party content (documents, emails) | Misinterpreting third-party content   | Gaining unauthorized access or control | Executing unintended commands or actions      |
 
-Azure AI Content Safety jailbreak risk detection recognizes four different classes of jailbreak attacks:  
+### Subtypes of User Prompt attacks
 
-|Category  |Description  |
-|---------|---------|
-|Attempt to change system rules   |    This category comprises, but is not limited to, requests to use a new unrestricted system/AI assistant without rules, principles, or limitations, or requests instructing the AI to ignore, forget and disregard its rules, instructions, and previous turns. |
-|Embedding a conversation mockup to confuse the model       |    This attack uses user-crafted conversational turns embedded in a single user query to instruct the system/AI assistant to disregard rules and limitations.      |
-|Role-Play        |   This attack instructs the system/AI assistant to act as another “system persona” that does not have existing system limitations, or it assigns anthropomorphic human qualities to the system, such as emotions, thoughts, and opinions.       |
-|Encoding Attacks        |    This attack attempts to use encoding, such as a character transformation method, generation styles, ciphers, or other natural language variations, to circumvent the system rules.      |
+**Prompt Shields for User Prompt attacks** recognizes the following classes of attacks:
+
+| Category           | Description   |
+| :--------- | :------ |
+| **Attempt to change system rules**      | This category includes, but is not limited to, requests to use a new unrestricted system/AI assistant without rules, principles, or limitations, or requests instructing the AI to ignore, forget and disregard its rules, instructions, and previous turns. |
+| **Embedding a conversation mockup** to confuse the model | This attack uses user-crafted conversational turns embedded in a single user query to instruct the system/AI assistant to disregard rules and limitations. |
+| **Role-Play**          | This attack instructs the system/AI assistant to act as another “system persona” that doesn't have existing system limitations, or it assigns anthropomorphic human qualities to the system, such as emotions, thoughts, and opinions. |
+| **Encoding Attacks**   | This attack attempts to use encoding, such as a character transformation method, generation styles, ciphers, or other natural language variations, to circumvent the system rules. |
+
+### Subtypes of Document attacks
+
+**Prompt Shields for Documents attacks** recognizes the following classes of attacks:
+
+|Category      | Description   |
+| ------------ | ------- |
+| **Manipulated  Content**   | Commands related to falsifying, hiding, manipulating, or pushing  specific information. |
+| **Intrusion** | Commands related to creating backdoor, unauthorized privilege  escalation, and gaining access to LLMs and systems |
+| **Information  Gathering** | Commands related to deleting, modifying, or accessing data or  stealing data. |
+| **Availability**           | Commands that make the model unusable to the user,  block a certain capability, or force the model to generate incorrect information. |
+| **Fraud**     | Commands related to defrauding the user out of money, passwords,  information, or acting on behalf of the user without authorization |
+| **Malware**  | Commands related to spreading malware via malicious links,  emails, etc. |
+| **Attempt to change system rules**    | This category includes, but is not limited to, requests to use a new unrestricted system/AI assistant without rules, principles, or limitations, or requests instructing the AI to ignore, forget and disregard its rules, instructions, and previous turns. |
+| **Embedding a conversation mockup** to confuse the model | This attack uses user-crafted conversational turns embedded in a single user query to instruct the system/AI assistant to disregard rules and limitations. |
+| **Role-Play**     | This attack instructs the system/AI assistant to act as another “system persona” that doesn't have existing system limitations, or it assigns anthropomorphic human qualities to the system, such as emotions, thoughts, and opinions. |
+| **Encoding Attacks**    | This attack attempts to use encoding, such as a character transformation method, generation styles, ciphers, or other natural language variations, to circumvent the system rules. |
+
+## Limitations
+
+### Language availability
+
+Currently, the Prompt Shields API supports the English language. While our API doesn't restrict the submission of non-English content, we can't guarantee the same level of quality and accuracy in the analysis of such content. We recommend users to primarily submit content in English to ensure the most reliable and accurate results from the API.
+
+### Text length limitations
+
+The maximum character limit for Prompt Shields is 10,000 characters per API call, between both the user prompts and documents combines. If your input (either user prompts or documents) exceeds these character limitations, you'll encounter an error.
+
+### TPS limitations
+
+| Pricing Tier | Requests per 10 seconds |
+| :----------- | :---------------------------- |
+| F0           | 1000         |
+| S0           | 1000         |
+
+If you need a higher rate, please [contact us](mailto:[email protected]) to request it.
 
 ## Next steps
 
-Follow the how-to guide to get started using Azure AI Content Safety to detect jailbreak risk.
+Follow the quickstart to get started using Azure AI Content Safety to detect user input risks.
 
 > [!div class="nextstepaction"]
-> [Detect jailbreak risk](../quickstart-jailbreak.md)
+> [Prompt Shields quickstart](../quickstart-jailbreak.md)

articles/ai-services/content-safety/index.yml

@@ -57,22 +57,26 @@ landingContent:
         links:
           - text: Harm categories
             url: concepts/harm-categories.md
+          - text: Groundedness detection
+            url: concepts/groundedness.md
       - linkListType: quickstart
         links:
           - text: Using Content Safety Studio
             url: studio-quickstart.md?pivots=content-safety-studio
           - text: Using the REST API or client SDKs
             url: quickstart-text.md?pivots=programming-language-rest
+          - text: Detect groundedness in LLM responses
+            url: quickstart-groundedness.md
       - linkListType: how-to-guide
         links:
           - text: Use a blocklist
             url: how-to/use-blocklist.md
 
-  - title: Jailbreak risk detection
+  - title: User input risk detection
     linkLists:
       - linkListType: concept
         links:
-          - text: Jailbreak risk detection
+          - text: Prompt Shields
             url: concepts/jailbreak-detection.md
       - linkListType: quickstart
         links:

articles/ai-services/content-safety/media/assigned-roles-simple.png

@@ -47,8 +47,9 @@ There are different types of analysis available from this service. The following
 | :-------------------------- | :---------------------- |
 | Analyze text API          | Scans text for sexual content, violence, hate, and self harm with multi-severity levels. |
 | Analyze image API         | Scans images for sexual content, violence, hate, and self harm with multi-severity levels. |
-| Jailbreak risk detection (new) | Scans text for the risk of a [jailbreak attack](./concepts/jailbreak-detection.md) on a Large Language Model. [Quickstart](./quickstart-jailbreak.md) |
-| Protected material text detection (new) | Scans AI-generated text for known text content (for example, song lyrics, articles, recipes, selected web content). [Quickstart](./quickstart-protected-material.md)|
+| Prompt Shields (new) | Scans text for the risk of a [User input attack](./concepts/jailbreak-detection.md) on a Large Language Model. [Quickstart](./quickstart-jailbreak.md) |
+| Groundedness detection (new) | Detects whether the text responses of large language models (LLMs) are grounded in the source materials provided by the users. [Quickstart](./quickstart-groundedness.md) |
+| Protected material text detection | Scans AI-generated text for known text content (for example, song lyrics, articles, recipes, selected web content). [Quickstart](./quickstart-protected-material.md)|
 
 ## Content Safety Studio
 
@@ -124,7 +125,7 @@ To use the Content Safety APIs, you must create your Azure AI Content Safety res
 - West US 2
 - Sweden Central
 
-Private preview features, such as jailbreak risk detection and protected material detection, are available in the following Azure regions:
+Public preview features, such as Prompt Shields and protected material detection, are available in the following Azure regions:
 - East US
 - West Europe