Skip to content

Commit 1961c2c

Browse files
Merge pull request #284609 from PatrickFarley/content-safety-updates
Content safety updates
2 parents e0872c6 + 7d5c1ff commit 1961c2c

11 files changed

+80
-71
lines changed

articles/ai-services/content-safety/concepts/jailbreak-detection.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ This shield aims to safeguard against attacks that use information not directly
8686

8787
### Language availability
8888

89-
Prompt Shields have been specifically trained and tested on the following languages: Chinese, English, French, German, Italian, Japanese, Portuguese. However, the feature can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.
89+
Prompt Shields have been specifically trained and tested on the following languages: Chinese, English, French, German, Spanish, Italian, Japanese, Portuguese. However, the feature can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.
9090

9191
### Text length limitations
9292

articles/ai-services/content-safety/how-to/custom-categories-rapid.md

Lines changed: 19 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -52,10 +52,7 @@ The following command creates an incident with a name and definition.
5252
curl --location --request PATCH 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
5353
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
5454
--header 'Content-Type: application/json' \
55-
--data '{
56-
\"incidentName\": \"<text-incident-name>\",
57-
\"incidentDefinition\": \"string\"
58-
}'
55+
--data '{ \"incidentName\": \"<test-incident>\", \"incidentDefinition\": \"<string>\"}'
5956
```
6057

6158
#### [Python](#tab/python)
@@ -115,9 +112,9 @@ curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-
115112
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
116113
--header 'Content-Type: application/json' \
117114
--data-raw '{
118-
"IncidentSamples": [
119-
{ "text": "<text-example-1>"},
120-
{ "text": "<text-example-2>"},
115+
\"IncidentSamples\": [
116+
{ \"text\": \"<text-example-1>\"},
117+
{ \"text\": \"<text-example-2>\"},
121118
...
122119
]
123120
}'
@@ -198,9 +195,9 @@ curl --location 'https://<endpoint>/contentsafety/text:detectIncidents?api-versi
198195
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
199196
--header 'Content-Type: application/json' \
200197
--data '{
201-
"text": "<test-text>",
202-
"incidentNames": [
203-
"<text-incident-name>"
198+
\"text\": \"<test-text>\",
199+
\"incidentNames\": [
200+
\"<text-incident-name>\"
204201
]
205202
}'
206203
```
@@ -309,11 +306,11 @@ curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-inciden
309306
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
310307
--header 'Content-Type: application/json' \
311308
--data '{
312-
"IncidentSamples": [
309+
\"IncidentSamples\": [
313310
{
314-
"image": {
315-
"content": "<base64-data>",
316-
"bloburl": "<your-blob-storage-url>.png"
311+
\"image\": {
312+
\"content\": \"<base64-data>\",
313+
\"bloburl\": \"<your-blob-storage-url>.png\"
317314
}
318315
}
319316
]
@@ -394,12 +391,12 @@ curl --location 'https://<endpoint>/contentsafety/image:detectIncidents?api-vers
394391
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
395392
--header 'Content-Type: application/json' \
396393
--data '{
397-
"image": {
398-
"url": "<your-blob-storage-url>/image.png",
394+
\"image\": {
395+
\"url\": \"<your-blob-storage-url>/image.png\",
399396
"content": "<base64-data>"
400397
},
401-
"incidentNames": [
402-
"<image-incident-name>"
398+
\"incidentNames\": [
399+
\"<image-incident-name>\"
403400
]
404401
}
405402
}'
@@ -588,8 +585,8 @@ curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-
588585
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
589586
--header 'Content-Type: application/json' \
590587
--data '{
591-
"IncidentSampleIds": [
592-
"<your-incident-sample-id>"
588+
\"IncidentSampleIds\": [
589+
\"<your-incident-sample-id>\"
593590
]
594591
}'
595592
```
@@ -769,8 +766,8 @@ curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-inciden
769766
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
770767
--header 'Content-Type: application/json' \
771768
--data '{
772-
"IncidentSampleIds": [
773-
"<your-incident-sample-id>"
769+
\"IncidentSampleIds\": [
770+
\"<your-incident-sample-id>\"
774771
]
775772
}'
776773
```

articles/ai-services/content-safety/language-support.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ ms.author: pafarley
1717
> [!IMPORTANT]
1818
> The Azure AI Content Safety models for protected material, groundedness detection, and custom categories (standard) work with English only.
1919
>
20-
> Other Azure AI Content Safety models have been specifically trained and tested on the following languages: Chinese, English, French, German, Italian, Japanese, Portuguese. However, these features can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.
20+
> Other Azure AI Content Safety models have been specifically trained and tested on the following languages: Chinese, English, French, German, Spanish, Italian, Japanese, Portuguese. However, these features can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.
2121
2222
> [!NOTE]
2323
> **Language auto-detection**

articles/ai-services/content-safety/overview.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,9 @@ Content filtering software can help your app comply with regulations or maintain
2222

2323
This documentation contains the following article types:
2424

25+
* **[Concepts](concepts/harm-categories.md)** provide in-depth explanations of the service functionality and features.
2526
* **[Quickstarts](./quickstart-text.md)** are getting-started instructions to guide you through making requests to the service.
2627
* **[How-to guides](./how-to/use-blocklist.md)** contain instructions for using the service in more specific or customized ways.
27-
* **[Concepts](concepts/harm-categories.md)** provide in-depth explanations of the service functionality and features.
2828

2929
## Where it's used
3030

@@ -41,19 +41,19 @@ The following are a few scenarios in which a software developer or team would re
4141
> [!IMPORTANT]
4242
> You cannot use Azure AI Content Safety to detect illegal child exploitation images.
4343
44-
## Product types
44+
## Product features
4545

4646
There are different types of analysis available from this service. The following table describes the currently available APIs.
4747

48-
| Type | Functionality |
49-
| :-------------------------- | :---------------------- |
50-
| [Prompt Shields](/rest/api/contentsafety/text-operations/detect-text-jailbreak) (preview) | Scans text for the risk of a [User input attack](./concepts/jailbreak-detection.md) on a Large Language Model. [Quickstart](./quickstart-jailbreak.md) |
51-
| [Groundedness detection](/rest/api/contentsafety/text-groundedness-detection-operations/detect-groundedness-options) (preview) | Detects whether the text responses of large language models (LLMs) are grounded in the source materials provided by the users. [Quickstart](./quickstart-groundedness.md) |
52-
| [Protected material text detection](/rest/api/contentsafety/text-operations/detect-text-protected-material) (preview) | Scans AI-generated text for [known text content](./concepts/protected-material.md) (for example, song lyrics, articles, recipes, selected web content). [Quickstart](./quickstart-protected-material.md)|
53-
| Custom categories API (preview) | Lets you create and train your own [custom content categories](./concepts/custom-categories.md) and scan text for matches. [Quickstart](./quickstart-custom-categories.md) |
54-
| Custom categories (rapid) API (preview) | Lets you define [emerging harmful content patterns](./concepts/custom-categories.md) and scan text and images for matches. [How-to guide](./how-to/custom-categories-rapid.md) |
55-
| [Analyze text](/rest/api/contentsafety/text-operations/analyze-text) API | Scans text for sexual content, violence, hate, and self harm with multi-severity levels. |
56-
| [Analyze image](/rest/api/contentsafety/image-operations/analyze-image) API | Scans images for sexual content, violence, hate, and self harm with multi-severity levels. |
48+
| Feature | Functionality | Concepts guide | Get started |
49+
| :-------------------------- | :---------------------- | --| --|
50+
| [Prompt Shields](/rest/api/contentsafety/text-operations/detect-text-jailbreak) (preview) | Scans text for the risk of a User input attack on a Large Language Model. | [Prompt Shields concepts](/azure/ai-services/content-safety/concepts/jailbreak-detection)|[Quickstart](./quickstart-jailbreak.md) |
51+
| [Groundedness detection](/rest/api/contentsafety/text-groundedness-detection-operations/detect-groundedness-options) (preview) | Detects whether the text responses of large language models (LLMs) are grounded in the source materials provided by the users. | [Groundedness detection concepts](/azure/ai-services/content-safety/concepts/groundedness)|[Quickstart](./quickstart-groundedness.md) |
52+
| [Protected material text detection](/rest/api/contentsafety/text-operations/detect-text-protected-material) (preview) | Scans AI-generated text for known text content (for example, song lyrics, articles, recipes, selected web content). | [Protected material concepts](/azure/ai-services/content-safety/concepts/protected-material)|[Quickstart](./quickstart-protected-material.md)|
53+
| Custom categories API (preview) | Lets you create and train your own custom content categories and scan text for matches. | [Custom categories concepts](/azure/ai-services/content-safety/concepts/custom-categories)|[Quickstart](./quickstart-custom-categories.md) |
54+
| Custom categories (rapid) API (preview) | Lets you define emerging harmful content patterns and scan text and images for matches. | [Custom categories concepts](/azure/ai-services/content-safety/concepts/custom-categories)| [How-to guide](./how-to/custom-categories-rapid.md) |
55+
| [Analyze text](/rest/api/contentsafety/text-operations/analyze-text) API | Scans text for sexual content, violence, hate, and self harm with multi-severity levels. | [Harm categories](/azure/ai-services/content-safety/concepts/harm-categories)| [Quickstart](/azure/ai-services/content-safety/quickstart-text) |
56+
| [Analyze image](/rest/api/contentsafety/image-operations/analyze-image) API | Scans images for sexual content, violence, hate, and self harm with multi-severity levels. | [Harm categories](/azure/ai-services/content-safety/concepts/harm-categories)| [Quickstart](/azure/ai-services/content-safety/quickstart-image) |
5757

5858

5959
## Content Safety Studio
@@ -130,7 +130,7 @@ See the following list for the input requirements for each feature.
130130

131131
### Language support
132132

133-
Content Safety models have been specifically trained and tested in the following languages: English, German, Japanese, Spanish, French, Italian, Portuguese, and Chinese. However, the service can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.
133+
Content Safety models have been specifically trained and tested in the following languages: English, German, Spanish, Japanese, French, Italian, Portuguese, and Chinese. However, the service can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.
134134

135135
Custom Categories currently only works well in English. You can try to use other languages with your own dataset, but the quality might vary across languages.
136136

articles/ai-services/content-safety/quickstart-custom-categories.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ ms.author: pafarley
1515

1616
Follow this guide to use Azure AI Content Safety Custom category REST API to create your own content categories for your use case and train Azure AI Content Safety to detect them in new text content.
1717

18+
For more information on Custom categories, see the [Custom categories concept page](./concepts/custom-categories.md). For API input limits, see the [Input requirements](./overview.md#input-requirements) section of the Overview.
19+
1820
> [!IMPORTANT]
1921
> This feature is only available in certain Azure regions. See [Region availability](./overview.md#region-availability).
2022

articles/ai-services/content-safety/quickstart-groundedness.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ ms.author: pafarley
1515

1616
Follow this guide to use Azure AI Content Safety Groundedness detection to check whether the text responses of large language models (LLMs) are grounded in the source materials provided by the users.
1717

18+
For more information on Groundedness detection, see the [Groundedness detection concept page](./concepts/groundedness.md). For API input limits, see the [Input requirements](./overview.md#input-requirements) section of the Overview.
19+
1820
## Prerequisites
1921

2022
* An Azure subscription - [Create one for free](https://azure.microsoft.com/free/cognitive-services/)
@@ -339,9 +341,8 @@ If you want to clean up and remove an Azure AI services subscription, you can de
339341
- [Azure portal](/azure/ai-services/multi-service-resource?pivots=azportal#clean-up-resources)
340342
- [Azure CLI](/azure/ai-services/multi-service-resource?pivots=azcli#clean-up-resources)
341343

342-
## Next steps
344+
## Related content
343345

344-
Combine Groundedness detection with other LLM safety features like Prompt Shields.
346+
* [Groundedness detection concepts](./concepts/groundedness.md)
347+
* Combine Groundedness detection with other LLM safety features like [Prompt Shields](./quickstart-jailbreak.md).
345348

346-
> [!div class="nextstepaction"]
347-
> [Prompt Shields quickstart](./quickstart-jailbreak.md)

articles/ai-services/content-safety/quickstart-image.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ zone_pivot_groups: programming-languages-content-safety
1717

1818
Get started with the Content Studio, REST API, or client SDKs to do basic image moderation. The Azure AI Content Safety service provides you with AI algorithms for flagging objectionable content. Follow these steps to try it out.
1919

20+
For more information on image moderation, see the [Harm categories concept page](./concepts/harm-categories.md). For API input limits, see the [Input requirements](./overview.md#input-requirements) section of the Overview.
21+
2022
> [!NOTE]
2123
>
2224
> The sample data and code may contain offensive content. User discretion is advised.
@@ -60,9 +62,8 @@ If you want to clean up and remove an Azure AI services subscription, you can de
6062
- [Azure portal](../multi-service-resource.md?pivots=azportal#clean-up-resources)
6163
- [Azure CLI](../multi-service-resource.md?pivots=azcli#clean-up-resources)
6264

63-
## Next steps
65+
## Related content
6466

65-
Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.
67+
* [Harm categories](./concepts/harm-categories.md)
68+
* Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.
6669

67-
> [!div class="nextstepaction"]
68-
> [Content Safety Studio quickstart](./studio-quickstart.md)

articles/ai-services/content-safety/quickstart-jailbreak.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ ms.author: pafarley
1515

1616
Follow this guide to use Azure AI Content Safety Prompt Shields to check your large language model (LLM) inputs for both User Prompt and Document attacks.
1717

18+
For more information on Prompt Shields, see the [Prompt Shields concept page](./concepts/jailbreak-detection.md). For API input limits, see the [Input requirements](./overview.md#input-requirements) section of the Overview.
19+
1820
## Prerequisites
1921

2022
* An Azure subscription - [Create one for free](https://azure.microsoft.com/free/cognitive-services/)
@@ -94,9 +96,8 @@ If you want to clean up and remove an Azure AI services subscription, you can de
9496
- [Azure portal](/azure/ai-services/multi-service-resource?pivots=azportal#clean-up-resources)
9597
- [Azure CLI](/azure/ai-services/multi-service-resource?pivots=azcli#clean-up-resources)
9698

97-
## Next steps
9899

99-
Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.
100+
## Related content
100101

101-
> [!div class="nextstepaction"]
102-
> [Content Safety Studio quickstart](./studio-quickstart.md)
102+
* [Prompt Shields concepts](./concepts/jailbreak-detection.md)
103+
* Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.

articles/ai-services/content-safety/quickstart-protected-material.md

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@ ms.author: pafarley
1515

1616
Protected material text describes language that matches known text content (for example, song lyrics, articles, recipes, selected web content). This feature can be used to identify and block known text content from being displayed in language model output (English content only). For more information, see [Protected material concepts](./concepts/protected-material.md).
1717

18+
For more information on protected material detection, see the [Protected material detection concept page](./concepts/protected-material.md). For API input limits, see the [Input requirements](./overview.md#input-requirements) section of the Overview.
19+
20+
1821
## Prerequisites
1922

2023
* An Azure subscription - [Create one for free](https://azure.microsoft.com/free/cognitive-services/)
@@ -87,9 +90,9 @@ If you want to clean up and remove an Azure AI services subscription, you can de
8790
- [Azure portal](../multi-service-resource.md?pivots=azportal#clean-up-resources)
8891
- [Azure CLI](../multi-service-resource.md?pivots=azcli#clean-up-resources)
8992

90-
## Next steps
9193

92-
Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.
94+
## Related content
95+
96+
* [Protected material detection concepts](./concepts/protected-material.md)
97+
* Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.
9398

94-
> [!div class="nextstepaction"]
95-
> [Content Safety Studio quickstart](./studio-quickstart.md)

articles/ai-services/content-safety/quickstart-text.md

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ zone_pivot_groups: programming-languages-content-safety
1717

1818
Get started with the Content Safety Studio, REST API, or client SDKs to do basic text moderation. The Azure AI Content Safety service provides you with AI algorithms for flagging objectionable content. Follow these steps to try it out.
1919

20+
For more information on text moderation, see the [Harm categories concept page](./concepts/harm-categories.md). For API input limits, see the [Input requirements](./overview.md#input-requirements) section of the Overview.
21+
22+
2023
> [!NOTE]
2124
>
2225
> The sample data and code may contain offensive content. User discretion is advised.
@@ -60,8 +63,8 @@ If you want to clean up and remove an Azure AI services subscription, you can de
6063
- [Azure portal](../multi-service-resource.md?pivots=azportal#clean-up-resources)
6164
- [Azure CLI](../multi-service-resource.md?pivots=azcli#clean-up-resources)
6265

63-
## Next steps
64-
Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.
6566

66-
> [!div class="nextstepaction"]
67-
> [Content Safety Studio quickstart](./studio-quickstart.md)
67+
## Related content
68+
69+
* [Harm categories](./concepts/harm-categories.md)
70+
* Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.

0 commit comments

Comments
 (0)