Merge pull request #274526 from PatrickFarley/content-safety-build

Content safety build

Author: v-dirichards

articles/ai-services/content-safety/concepts/custom-categories.md

@@ -0,0 +1,77 @@
+---
+title: "Custom categories in Azure AI Content Safety"
+titleSuffix: Azure AI services
+description: Learn about custom content categories and how you can use Azure AI Content Safety to handle them on your platform.
+#services: cognitive-services
+author: PatrickFarley
+manager: nitinme
+ms.service: azure-ai-content-safety
+ms.custom: build-2024
+ms.topic: conceptual
+ms.date: 04/11/2024
+ms.author: pafarley
+---
+
+# Custom categories 
+
+The Azure AI Content Safety custom category feature lets you create and manage your own content categories for enhanced moderation and filtering. This feature enables customers to define categories specific to their needs, provide sample data, train a custom machine learning model, and use it to classify new content according to the predefined categories.
+
+## Types of analysis
+
+| API        | Functionality   |
+| :--------- | :------------ |
+| Customized categories | Create, get, and delete a customized category or list all customized categories for further annotation tasks |
+
+## How it works
+
+The Azure AI Content Safety custom category feature provides a streamlined process for creating, training, and using custom content classification models. Here's a look at the underlying workflow:
+
+### Step 1: Definition and setup
+ 
+When you define a custom category, you need to teach the AI what type of content you want to identify. This involves providing a clear **category name** and a detailed **definition** that encapsulates the content's characteristics.
+
+Then, you collect a balanced dataset with **positive** and (optionally) **negative examples** to help the AI to learn the nuances of your category. This data should be representative of the variety of content that the model will encounter in a real-world scenario.
+
+### Step 2: Model training
+ 
+Once your dataset is ready, the Azure AI Content Safety service uses it to train a new model. During training, the AI analyzes the data and learns to distinguish between content that matches the category and content that doesn't.
+
+### Step 3: Model inferencing
+ 
+After training, you need to evaluate the model to ensure it meets your accuracy requirements. Test the model with new content that it hasn't received before. The evaluation phase helps you identify any potential adjustments you need to make deploying the model into a production environment.
+
+## Limitations
+
+### Input limitations
+
+See the following table for the input limitations of the service.
+
+| Object           | Limitation   |
+| ---------------- | ------------ |
+| Supported languages | English only |
+|     Number of categories per user     |         5     |
+|         Number of versions per category   |        5      |
+|       Number of concurrent builds (processes) per category      |       1       |
+|       Inference operations per second           |    10          |
+|        Number of custom categories in one text analyze request          |       5       |
+|        Number of samples in a category version          |        minimum 50, maximum 10K (no duplicate samples allowed)      |
+|       Sample file size       |     maximum 128000 bytes         |
+|       Length of a text sample           |          maximum 125K characters   |
+|        Length of a category definition          |       maximum 1000 characters     |
+|       Length of a category name           |         maximum 128 characters    |
+|       Length of a blob url       |          maximum 500 characters    |
+
+
+### Region availability
+
+To use this API, you must create your Azure AI Content Safety resource in one of the supported regions:
+- East US
+
+
+## Next steps
+
+Follow the how-to guide to create custom categories in Azure AI Content Safety.
+
+* [Use custom category API](../how-to/custom-categories.md)
+
+

articles/ai-services/content-safety/concepts/incident-response.md

@@ -0,0 +1,61 @@
+---
+title: "Incident response in Azure AI Content Safety"
+titleSuffix: Azure AI services
+description: Learn about content incidents and how you can use Azure AI Content Safety to handle them on your platform.
+#services: cognitive-services
+author: PatrickFarley
+manager: nitinme
+ms.service: azure-ai-content-safety
+ms.custom: build-2024
+ms.topic: conceptual
+ms.date: 04/11/2024
+ms.author: pafarley
+---
+
+# Incident response
+
+In content moderation scenarios, incident response is the process of identifying, analyzing, containing, eradicating, and recovering from cyber incidents that involve inappropriate or harmful content on online platforms. 
+
+An incident may involve a set of emerging content patterns (text, image, or other modalities) that violate Microsoft community guidelines or the customers' own policies and expectations. These incidents need to be mitigated quickly and accurately to avoid potential live site issues or harm to users and communities. 
+
+## Incident response API features
+
+One way to deal with emerging content incidents is to use [Blocklists](/azure/ai-services/content-safety/how-to/use-blocklist), but that only allows exact text matching and no image matching. The Azure AI Content Safety incident response API offers the following advanced capabilities:
+- semantic text matching using embedding search with a lightweight classifier
+- image matching with a lightweight object-tracking model and embedding search.
+
+## How it works 
+
+First, you use the API to create an incident object with a description. Then you add any number of image or text samples to the incident. No training step is needed.
+
+Then, you can include your defined incident in a regular text analysis or image analysis request. The service will indicate whether the submitted content is an instance of your incident. The service can still do other content moderation tasks in the same API call.
+
+## Limitations
+
+### Language availability
+
+The text incident response API supports all languages that are supported by Content Safety text moderation. See [Language support](/azure/ai-services/content-safety/language-support).
+
+### Input limitations
+
+See the following table for the input limitations of the incident response API:
+
+| Object     | Limitation      |
+| :------------ | :----------- |
+| Maximum length of an incident name | 100 characters | 
+| Maximum number of text/image samples per incident | 1000 |
+| Maximum size of each sample | Text: 500 characters<br>Image: 4 MB  |
+| Maximum number of text or image incidents per resource| 100 |  
+| Supported Image formats | BMP, GIF, JPEG, PNG, TIF, WEBP |
+
+### Region availability
+
+To use this API, you must create your Azure AI Content Safety resource in one of the supported regions:
+- East US
+- Sweden Central
+
+## Next steps
+
+Follow the how-to guide to use the Azure AI Content Safety incident response API.
+
+* [Use the incident response API](../how-to/incident-response.md)

articles/ai-services/content-safety/faq.yml

@@ -37,6 +37,10 @@ sections:
           - Azure AI Content Safety offers more accurate and granular detection of harmful content in text and images using state-of-the-art AI models.
           - Azure AI Content Safety supports multilingual content moderation in English, Japanese, German, Spanish, French, Portuguese, Italian, and Chinese.
           - Azure AI Content Safety enables responsible AI practices by monitoring both user-generated and AI-generated content.
+      - question: |
+          Can I detect harmful content in custom categories that I define myself?
+        answer: |
+          Yes, you can use the Azure AI Content Safety custom categories API to create categories, upload examples, and train the service to detect similar content. Custom categories are available for both text and image moderation.
       - question: |
           How does billing work for the Azure AI Content Safety service?
         answer: |