You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Azure AI Video Indexer face detection overview
2
+
title: Face detection overview
3
3
titleSuffix: Azure AI Video Indexer
4
-
description: This article gives an overview of an Azure AI Video Indexer face detection.
4
+
description: Get an overview of face detection in Azure AI Video Indexer.
5
5
author: juliako
6
6
ms.author: juliako
7
7
manager: femila
@@ -12,45 +12,49 @@ ms.topic: article
12
12
13
13
# Face detection
14
14
15
-
> [!IMPORTANT]
16
-
> Face identification, customization and celebrity recognition features access is limited based on eligibility and usage criteria in order to support our Responsible AI principles. Face identification, customization and celebrity recognition features are only available to Microsoft managed customers and partners. Use the [Face Recognition intake form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUQjA5SkYzNDM4TkcwQzNEOE1NVEdKUUlRRCQlQCN0PWcu) to apply for access.
15
+
Face detection, a feature of Azure AI Video Indexer, automatically detects faces in a media file, and then aggregates instances of similar faces into groups. The celebrities recognition model then runs to recognize celebrities.
16
+
17
+
The celebrities recognition model covers approximately 1 million faces and is based on commonly requested data sources. Faces that Video Indexer doesn't recognize as celebrities are still detected but are left unnamed. You can build your own custom [person model](/azure/azure-video-indexer/customize-person-model-overview) to train Video Indexer to recognize faces that aren't recognized by default.
17
18
18
-
Face detection is an Azure AI Video Indexer AI feature that automatically detects faces in a media file and aggregates instances of similar faces into the same group. The celebrities recognition module is then run to recognize celebrities. This module covers approximately one million faces and is based on commonly requested data sources. Faces that aren't recognized by Azure AI Video Indexer are still detected but are left unnamed. Customers can build their own custom [Person modules](/azure/azure-video-indexer/customize-person-model-overview) whereby the Azure AI Video Indexer recognizes faces that aren't recognized by default.
19
+
Face detection insights are generated as a categorized list in a JSON file that includes a thumbnail and either a name or an ID for each face. Selecting a face’s thumbnail displays information like the name of the person (if they were recognized), the percentage of the video that the person appears, and the person's biography, if they're a celebrity. You can also scroll between instances in the video where the person appears.
19
20
20
-
The resulting insights are generated in a categorized list in a JSON file that includes a thumbnail and either name or ID of each face. Clicking face’s thumbnail displays information like the name of the person (if they were recognized), the % of appearances in the video, and their biography if they're a celebrity. It also enables scrolling between the instances in the video.
21
+
> [!IMPORTANT]
22
+
> To support Microsoft Responsible AI principles, access to face identification, customization, and celebrity recognition features is limited and based on eligibility and usage criteria. Face identification, customization, and celebrity recognition features are available to Microsoft managed customers and partners. To apply for access, use the [facial recognition intake form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUQjA5SkYzNDM4TkcwQzNEOE1NVEdKUUlRRCQlQCN0PWcu).
Review [Transparency Note for Azure AI Video Indexer](/legal/azure-video-indexer/transparency-note?context=/azure/azure-video-indexer/context/context).
25
27
26
-
## General principles
28
+
## General principles
27
29
28
-
This article discusses faces detection and the key considerations for making use of this technology responsibly. There are many things you need to consider when deciding how to use and implement an AI-powered feature:
30
+
This article discusses face detection and key considerations for using this technology responsibly. You need to consider many important factors when you decide how to use and implement an AI-powered feature, including:
29
31
30
-
- Will this feature perform well in my scenario? Before deploying faces detection into your scenario, test how it performs using real-life data and make sure it can deliver the accuracy you need.
31
-
- Are we equipped to identify and respond to errors? AI-powered products and features won't be 100% accurate, so consider how you'll identify and respond to any errors that may occur.
32
+
- Will this feature perform well in your scenario? Before you deploy face detection in your scenario, test how it performs by using real-life data. Make sure that it can deliver the accuracy you need.
33
+
- Are you equipped to identify and respond to errors? AI-powered products and features aren't 100 percent accurate, so consider how you'll identify and respond to any errors that occur.
32
34
33
35
## Key terms
34
36
35
-
|Term|Definition|
37
+
|Term|Definition|
36
38
|---|---|
37
-
|Insight|The information and knowledge derived from the processing and analysis of video and audio files that generate different types of insights and can include detected objects, people, faces, keyframes and translations or transcriptions. |
38
-
|Face recognition |The analysis of images to identify the faces that appear in the images. This process is implemented via the Azure AI Face API. |
39
-
|Template |Enrolled images of people are converted to templates, which are then used for facial recognition. Machine-interpretable features are extracted from one or more images of an individual to create that individual’s template. The enrollment or probe images aren't stored by Face API and the original images can't be reconstructed based on a template. Template quality is a key determinant on the accuracy of your results. |
40
-
|Enrollment |The process of enrolling images of individuals for template creation so they can be recognized. When a person is enrolled to a verification system used for authentication, their template is also associated with a primary identifier2 that is used to determine which template to compare with the probe template. High-quality images and images representing natural variations in how a person looks (for instance wearing glasses, not wearing glasses) generate high-quality enrollment templates. |
41
-
|Deep search |The ability to retrieve only relevant video and audio files from a video library by searching for specific terms within the extracted insights.|
39
+
| insight|The information and knowledge that you derive from processing and analyzing video and audio files. The insight can include detected objects, people, faces, keyframes, and translations or transcriptions. |
40
+
| face recognition | Analyzing images to identify the faces that appear in the images. This process is implemented via the Azure AI Face API. |
41
+
| template |Enrolled images of people are converted to templates, which are then used for facial recognition. Machine-interpretable features are extracted from one or more images of an individual to create that individual’s template. The enrollment or probe images aren't stored by the Face API, and the original images can't be reconstructed based on a template. Template quality is a key determinant for accuracy in your results. |
42
+
| enrollment |The process of enrolling images of individuals for template creation so that they can be recognized. When a person is enrolled to a verification system that's used for authentication, their template is also associated with a primary identifier that's used to determine which template to compare against the probe template. High-quality images and images that represent natural variations in how a person looks (for instance, wearing glasses and not wearing glasses) generate high-quality enrollment templates. |
43
+
| deep search |The ability to retrieve only relevant video and audio files from a video library by searching for specific terms within the extracted insights.|
42
44
43
-
## View the insight
45
+
## View insights
44
46
45
-
To see the instances on the website, do the following:
47
+
To see face detection instances on the Azure AI Video Indexer website:
46
48
47
-
1. When uploading the media file, go to Video + Audio Indexing, or go to Audio Only or Video + Audio and select Advanced.
48
-
1. After the file is uploaded and indexed, go to Insights and scroll to People.
49
+
1. When you upload the media file, in the **Upload and index** dialog, select **Advanced settings**.
50
+
1. On the left menu, select **People models**. Select a model to apply to the media file.
51
+
1. After the file is uploaded and indexed, go to **Insights** and scroll to **People**.
49
52
50
-
To see face detection insight in the JSON file, do the following:
53
+
To see face detection insights in a JSON file:
51
54
52
-
1. Select Download -> Insights (JSON).
53
-
1. Copy the `faces` element, under `insights`, and paste it into your JSON viewer.
55
+
1. On the Azure AI Video Indexer website, open the uploaded video.
56
+
1. Select **Download** > **Insights (JSON)**.
57
+
1. Under `insights`, copy the `faces` element and paste it into your JSON viewer.
54
58
55
59
```json
56
60
"faces": [
@@ -93,67 +97,64 @@ To see face detection insight in the JSON file, do the following:
93
97
]
94
98
```
95
99
96
-
To download the JSON file via the API, [Azure AI Video Indexer developer portal](https://api-portal.videoindexer.ai/).
100
+
To download the JSON file via the API, go to the [Azure AI Video Indexer developer portal](https://api-portal.videoindexer.ai/).
97
101
98
102
> [!IMPORTANT]
99
-
> When reviewing face detections in the UI you may not see all faces, we expose only face groups with a confidence of more than 0.5 and the face must appear for a minimum of 4 seconds or 10% * video_duration. Only when these conditions are met we will show the face in the UI and the Insights.json. You can always retrieve all face instances from the Face Artifact file using the api `https://api.videoindexer.ai/{location}/Accounts/{accountId}/Videos/{videoId}/ArtifactUrl[?Faces][&accessToken]`
103
+
> When you review face detections in the UI, you might not see all faces that appear in the video. We expose only face groups that have a confidence of more than 0.5, and the face must appear for a minimum of 4 seconds or 10 percent of the value of `video_duration`. Only when these conditions are met do we show the face in the UI and in the *Insights.json* file. You can always retrieve all face instances from the face artifact file by using the API: `https://api.videoindexer.ai/{location}/Accounts/{accountId}/Videos/{videoId}/ArtifactUrl[?Faces][&accessToken]`.
100
104
101
-
## Face detection components
105
+
## Face detection components
102
106
103
-
During the Faces Detection procedure, images in a media file are processed, as follows:
107
+
The following table describes how images in a media file are processed during the face detection procedure:
104
108
105
-
|Component|Definition|
109
+
|Component | Definition|
106
110
|---|---|
107
-
|Source file | The user uploads the source file for indexing. |
108
-
|Detection and aggregation |The face detector identifies the faces in each frame. The faces are then aggregated and grouped. |
109
-
|Recognition |The celebrities module runs over the aggregated groups to recognize celebrities. If the customer has created their own **persons** module it's also run to recognize people. When people aren't recognized, they're labeled Unknown1, Unknown2 and so on. |
110
-
|Confidence value |Where applicable for well-known faces or faces identified in the customizable list, the estimated confidence level of each label is calculated as a range of 0 to 1. The confidence score represents the certainty in the accuracy of the result. For example, an 82% certainty is represented as an 0.82 score.|
111
+
| source file | The user uploads the source file for indexing. |
112
+
| detection and aggregation | The face detector identifies the faces in each frame. The faces are then aggregated and grouped. |
113
+
| recognition | The celebrities model processes the aggregated groups to recognize celebrities. If you've created your own people model, it also processes groups to recognize other people. If people aren't recognized, they're labeled Unknown1, Unknown2, and so on. |
114
+
| confidence value | Where applicable for well-known faces or for faces that are identified in the customizable list, the estimated confidence level of each label is calculated as a range of 0 to 1. The confidence score represents the certainty in the accuracy of the result. For example, an 82 percent certainty is represented as an 0.82 score.|
111
115
112
-
## Example use cases
116
+
## Example use cases
113
117
114
-
* Summarizing where an actor appears in a movie or reusing footage by deep searching for specific faces in organizational archives for insight on a specific celebrity.
115
-
* Improved efficiency when creating feature stories at a news or sports agency, for example deep searching for a celebrity or football player in organizational archives.
116
-
* Using faces appearing in the video to create promos, trailers or highlights. Azure AI Video Indexer can assist by adding keyframes, scene markers, timestamps and labeling so that content editors invest less time reviewing numerous files.
118
+
The following list describes examples of common use cases for face detection:
117
119
118
-
## Considerations when choosing a use case
120
+
- Summarize where an actor appears in a movie or reuse footage by deep searching specific faces in organizational archives for insight about a specific celebrity.
121
+
- Get improved efficiency when you create feature stories at a news agency or sports agency. Examples include deep searching a celebrity or a football player in organizational archives.
122
+
- Use faces that appear in a video to create promos, trailers, or highlights. Video Indexer can assist by adding keyframes, scene markers, time stamps, and labeling so that content editors invest less time reviewing numerous files.
119
123
120
-
* Carefully consider the accuracy of the results, to promote more accurate detections, check the quality of the video, low quality video might impact the detected insights.
121
-
* Carefully consider when using for law enforcement. People might not be detected if they're small, sitting, crouching, or obstructed by objects or other people. To ensure fair and high-quality decisions, combine face detection-based automation with human oversight.
122
-
* Don't use face detection for decisions that may have serious adverse impacts. Decisions based on incorrect output could have serious adverse impacts. Additionally, it's advisable to include human review of decisions that have the potential for serious impacts on individuals.
124
+
## Considerations for choosing a use case
123
125
124
-
When used responsibly and carefully face detection is a valuable tool for many industries. To respect the privacy and safety of others, and to comply with local and global regulations, we recommend the following:
126
+
Face detection is a valuable tool for many industries when it's used responsibly and carefully. To respect the privacy and safety of others, and to comply with local and global regulations, we recommend that you follow these use guidelines:
125
127
126
-
* Always respect an individual’s right to privacy, and only ingest videos for lawful and justifiable purposes.
127
-
* Don't purposely disclose inappropriate content about young children or family members of celebrities or other content that may be detrimental or pose a threat to an individual’s personal freedom.
128
-
* Commit to respecting and promoting human rights in the design and deployment of your analyzed media.
129
-
* When using third party materials, be aware of any existing copyrights or permissions required before distributing content derived from them.
130
-
* Always seek legal advice when using content from unknown sources.
131
-
* Always obtain appropriate legal and professional advice to ensure that your uploaded videos are secured and have adequate controls to preserve the integrity of your content and to prevent unauthorized access.
132
-
* Provide a feedback channel that allows users and individuals to report issues with the service.
133
-
* Be aware of any applicable laws or regulations that exist in your area regarding processing, analyzing, and sharing media containing people.
134
-
* Keep a human in the loop. Don't use any solution as a replacement for human oversight and decision-making.
135
-
* Fully examine and review the potential of any AI model you're using to understand its capabilities and limitations.
128
+
- Carefully consider the accuracy of the results. To promote more accurate detection, check the quality of the video. Low-quality video might affect the insights that are presented.
129
+
- Carefully review results if you use face detection for law enforcement. People might not be detected if they're small, sitting, crouching, or obstructed by objects or other people. To ensure fair and high-quality decisions, combine face detection-based automation with human oversight.
130
+
- Don't use face detection for decisions that might have serious, adverse impacts. Decisions that are based on incorrect output can have serious, adverse impacts. It's advisable to include human review of decisions that have the potential for serious impacts on individuals.
131
+
- Always respect an individual’s right to privacy, and ingest videos only for lawful and justifiable purposes.
132
+
- Don't purposely disclose inappropriate content about young children, family members of celebrities, or other content that might be detrimental to or pose a threat to an individual’s personal freedom.
133
+
- Commit to respecting and promoting human rights in the design and deployment of your analyzed media.
134
+
- If you use third-party materials, be aware of any existing copyrights or required permissions before you distribute content that's derived from them.
135
+
- Always seek legal advice if you use content from an unknown source.
136
+
- Always obtain appropriate legal and professional advice to ensure that your uploaded videos are secured and that they have adequate controls to preserve content integrity and prevent unauthorized access.
137
+
- Provide a feedback channel that allows users and individuals to report issues they might experience with the service.
138
+
- Be aware of any applicable laws or regulations that exist in your area about processing, analyzing, and sharing media that features people.
139
+
- Keep a human in the loop. Don't use any solution as a replacement for human oversight and decision making.
140
+
- Fully examine and review the potential of any AI model that you're using to understand its capabilities and limitations.
136
141
137
-
## Next steps
142
+
## Related content
138
143
139
-
### Learn More about Responsible AI
144
+
Learn more about Responsible AI:
140
145
141
-
- [Microsoft Responsible AI principles](https://www.microsoft.com/ai/responsible-ai?activetab=pivot1%3aprimaryr6)
146
+
- [Microsoft Responsible AI principles](https://www.microsoft.com/ai/responsible-ai?activetab=pivot1%3aprimaryr6)
142
147
- [Microsoft Responsible AI resources](https://www.microsoft.com/ai/responsible-ai-resources)
143
-
- [Microsoft Azure Learning courses on Responsible AI](/training/paths/responsible-ai-business-principles/)
148
+
- [Microsoft Azure Learn training courses on Responsible AI](/training/paths/responsible-ai-business-principles/)
144
149
- [Microsoft Global Human Rights Statement](https://www.microsoft.com/corporate-responsibility/human-rights-statement?activetab=pivot_1:primaryr5)
0 commit comments