|
| 1 | +--- |
| 2 | +title: PII Detection cognitive skill (preview) |
| 3 | +titleSuffix: Azure Cognitive Search |
| 4 | +description: Extract and mask personally identifiable information from text in an enrichment pipeline in Azure Cognitive Search. This skill is currently in public preview. |
| 5 | + |
| 6 | +manager: nitinme |
| 7 | +author: careyjmac |
| 8 | +ms.author: chalton |
| 9 | +ms.service: cognitive-search |
| 10 | +ms.topic: conceptual |
| 11 | +ms.date: 1/27/2020 |
| 12 | +--- |
| 13 | + |
| 14 | +# PII Detection cognitive skill |
| 15 | + |
| 16 | +> [!IMPORTANT] |
| 17 | +> This skill is currently in public preview. Preview functionality is provided without a service level agreement, and is not recommended for production workloads. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). |
| 18 | +> The [REST API version 2019-05-06-Preview](search-api-preview.md) provides preview features. There is currently no portal or .NET SDK support. |
| 19 | +
|
| 20 | +The **PII Detection** skill extracts personally identifiable information from a text and gives you the option to mask it in the text in various ways. This skill uses the machine learning models provided by [Text Analytics](https://docs.microsoft.com/azure/cognitive-services/text-analytics/overview) in Cognitive Services. |
| 21 | + |
| 22 | +> [!NOTE] |
| 23 | +> As you expand scope by increasing the frequency of processing, adding more documents, or adding more AI algorithms, you will need to [attach a billable Cognitive Services resource](cognitive-search-attach-cognitive-services.md). Charges accrue when calling APIs in Cognitive Services, and for image extraction as part of the document-cracking stage in Azure Cognitive Search. There are no charges for text extraction from documents. |
| 24 | +> |
| 25 | +> Execution of built-in skills is charged at the existing [Cognitive Services pay-as-you go price](https://azure.microsoft.com/pricing/details/cognitive-services/). Image extraction pricing is described on the [Azure Cognitive Search pricing page](https://go.microsoft.com/fwlink/?linkid=2042400). |
| 26 | +
|
| 27 | + |
| 28 | +## @odata.type |
| 29 | +Microsoft.Skills.Text.PIIDetectionSkill |
| 30 | + |
| 31 | +## Data limits |
| 32 | +The maximum size of a record should be 50,000 characters as measured by [`String.Length`](https://docs.microsoft.com/dotnet/api/system.string.length). If you need to break up your data before sending it to the skill, consider using the [Text Split skill](cognitive-search-skill-textsplit.md). |
| 33 | + |
| 34 | +## Skill parameters |
| 35 | + |
| 36 | +Parameters are case-sensitive and are all optional. |
| 37 | + |
| 38 | +| Parameter name | Description | |
| 39 | +|--------------------|-------------| |
| 40 | +| defaultLanguageCode | Language code of the input text. For now, only `en` is supported. | |
| 41 | +| minimumPrecision | A value between 0 and 1. If the confidence score (in the `piiEntities` output) is lower than this value, the entity is not returned or masked. The default is 0. | |
| 42 | +| maskingMode | A parameter that provides various ways to mask the detected PII in the input text. The following options are supported: <ul><li>`none` (default): This means that no masking will be performed and the `maskedText` output will not be returned. </li><li> `redact`: This option will remove the detected entities from the input text and not replace them with anything. Note that in this case, the offset in the `piiEntities` output will be in relation to the original text, and not the masked text. </li><li> `replace`: This option will replace the detected entities with the character given in the `maskingCharacter` parameter. The character will be repeated to the length of the detected entity so that the offsets will correctly correspond to both the input text as well as the output `maskedText`.</li></ul> | |
| 43 | +| maskingCharacter | The character that will be used to masked the text if the `maskingMode` parameter is set to `replace`. The following options are supported: `*` (default), `#`, `X`. This parameter can only be `null` if `maskingMode` is not set to `replace`. | |
| 44 | + |
| 45 | + |
| 46 | +## Skill inputs |
| 47 | + |
| 48 | +| Input name | Description | |
| 49 | +|---------------|-------------------------------| |
| 50 | +| languageCode | Optional. Default is `en`. | |
| 51 | +| text | The text to analyze. | |
| 52 | + |
| 53 | +## Skill outputs |
| 54 | + |
| 55 | +| Output name | Description | |
| 56 | +|---------------|-------------------------------| |
| 57 | +| piiEntities | An array of complex types that contains the following fields: <ul><li>text (The actual PII as extracted)</li> <li>type</li><li>subType</li><li>score (Higher value means it's more likely to be a real entity)</li><li>offset (into the input text)</li><li>length</li></ul> </br> [Possible types and subTypes can be found here.](https://docs.microsoft.com/azure/cognitive-services/text-analytics/named-entity-types?tabs=personal) | |
| 58 | +| maskedText | If `maskingMode` is set to a value other than `none`, this output will be the string result of the masking performed on the input text as described by the selected `maskingMode`. If `maskingMode` is set to `none`, this output will not be present. | |
| 59 | + |
| 60 | +## Sample definition |
| 61 | + |
| 62 | +```json |
| 63 | + { |
| 64 | + "@odata.type": "#Microsoft.Skills.Text.PIIDetectionSkill", |
| 65 | + "defaultLanguageCode": "en", |
| 66 | + "minimumPrecision": 0.5, |
| 67 | + "maskingMode": "replace", |
| 68 | + "maskingCharacter": "*", |
| 69 | + "inputs": [ |
| 70 | + { |
| 71 | + "name": "text", |
| 72 | + "source": "/document/content" |
| 73 | + } |
| 74 | + ], |
| 75 | + "outputs": [ |
| 76 | + { |
| 77 | + "name": "piiEntities" |
| 78 | + }, |
| 79 | + { |
| 80 | + "name": "maskedText" |
| 81 | + } |
| 82 | + ] |
| 83 | + } |
| 84 | +``` |
| 85 | +## Sample input |
| 86 | + |
| 87 | +```json |
| 88 | +{ |
| 89 | + "values": [ |
| 90 | + { |
| 91 | + "recordId": "1", |
| 92 | + "data": |
| 93 | + { |
| 94 | + "text": "Microsoft employee with ssn 859-98-0987 is using our awesome API's." |
| 95 | + } |
| 96 | + } |
| 97 | + ] |
| 98 | +} |
| 99 | +``` |
| 100 | + |
| 101 | +## Sample output |
| 102 | + |
| 103 | +```json |
| 104 | +{ |
| 105 | + "values": [ |
| 106 | + { |
| 107 | + "recordId": "1", |
| 108 | + "data" : |
| 109 | + { |
| 110 | + "piiEntities":[ |
| 111 | + { |
| 112 | + "text":"859-98-0987", |
| 113 | + "type":"U.S. Social Security Number (SSN)", |
| 114 | + "subtype":"", |
| 115 | + "offset":28, |
| 116 | + "length":11, |
| 117 | + "score":0.65 |
| 118 | + } |
| 119 | + ], |
| 120 | + "maskedText": "Microsoft employee with ssn *********** is using our awesome API's." |
| 121 | + } |
| 122 | + } |
| 123 | + ] |
| 124 | +} |
| 125 | +``` |
| 126 | + |
| 127 | + |
| 128 | +## Error cases |
| 129 | +If the language code for the document is unsupported, an error is returned and no entities are extracted. |
| 130 | + |
| 131 | +If the skill returns a warning for any reason, the output `maskedText` will be empty. This means that if you expect that output to exist for input into later skills, it will not work as intended. Keep this in mind when writing your skillset definition. |
| 132 | + |
| 133 | +## See also |
| 134 | + |
| 135 | ++ [Built-in skills](cognitive-search-predefined-skills.md) |
| 136 | ++ [How to define a skillset](cognitive-search-defining-skillset.md) |
0 commit comments