Skip to content

Commit 95a2fbb

Browse files
v-dirichardsv-dirichards
authored
Merge pull request #4670 from laujan/jan-4592-audio-overview
Jan 4592 audio overview
2 parents a5ffc6c + 5c93873 commit 95a2fbb

20 files changed

+233
-34
lines changed

‎articles/ai-services/content-understanding/audio/overview.md

Lines changed: 214 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,11 @@ title: Azure AI Content Understanding audio overview
33
titleSuffix: Azure AI services
44
description: Learn about Azure AI Content Understanding audio solutions
55
author: laujan
6-
ms.author: lajanuar
6+
ms.author: jagoerge
77
manager: nitinme
88
ms.service: azure-ai-content-understanding
99
ms.topic: overview
10-
ms.date: 03/18/2025
11-
ms.custom: ignite-2024-understanding-release
10+
ms.date: 05/19/2025
1211
---
1312

1413

@@ -33,16 +32,34 @@ Here are common scenarios for using Content Understanding with conversational au
3332
:::image type="content" source="../media/audio/overview/workflow-diagram.png" lightbox="../media/audio/overview/workflow-diagram.png" alt-text="Illustration of Content Understanding audio workflow.":::
3433

3534
Content Understanding serves as a cornerstone for Media Asset Management solutions, enabling the following capabilities for audio files:
36-
35+
3736
### Content extraction
3837

3938
* **Transcription**. Converts conversational audio into searchable and analyzable text-based transcripts in WebVTT format. Customizable fields can be generated from transcription data. Sentence-level and word-level timestamps are available upon request.
4039

41-
* **`Diarization`**. Distinguishes between speakers in a conversation, attributing parts of the transcript to specific speakers.
40+
> [!NOTE]
41+
>
42+
> Content Understanding supports the full set of [Azure AI Speech Speech to text languages](../../speech-service/language-support.md).
43+
> For languages with fast transcriptions support and for files ≤ 300 MB and/or ≤ 2 hours, transcription time is reduced substantially.
44+
45+
* **Diarization**. Distinguishes between speakers in a conversation, attributing parts of the transcript to specific speakers.
4246

4347
* **Speaker role detection**. Identifies agent and customer roles within contact center call data.
4448

45-
* **Language detection**. Automatically detects the language in the audio or uses specified language/locale hints.
49+
* **Multilingual transcription**. Generates multilingual transcripts, applying language/locale per phrase. Deviating from language detection this feature is enabled when no language/locale is specified or language is set to `auto`.
50+
51+
> [!NOTE]
52+
>
53+
> The following locales are currently supported:
54+
> * **Files ≤ 300 MB and/or ≤ 2 hours**: de-DE, en-AU, en-CA, en-GB, en-IN, en-US, es-ES, es-MX, fr-CA, fr-FR, hi-IN, it-IT, ja-JP, ko-KR, and zh-CN.
55+
> * **Files larger than 300 MB and/or longer than 4 hours**: en-US, es-ES, es-MX, fr-FR, hi-IN, it-IT, ja-JP, ko-KR, pt-BR, zh-CN.
56+
57+
* **Language detection**. Automatically detects the dominant language/locale which is used to transcribe the file. Set multiple languages/locales to enable language detection.
58+
59+
> [!NOTE]
60+
>
61+
> For files larger than 300 MB and/or longer than 2 hours and locales unsupported by Fast transcription, the file is processed generating a multilingual transcript based on the specified locales.
62+
> In case language detection fails, the first language/locale defined is used to transcribe the file.
4663
4764
### Field extraction
4865

@@ -59,15 +76,197 @@ Content Understanding offers advanced audio capabilities, including:
5976

6077
* **Scenario adaptability**. Adapt the service to your requirements by generating custom fields and extract relevant data.
6178

62-
## Content Understanding audio analyzer templates
63-
64-
Content Understanding offers customizable audio analyzer templates:
65-
66-
* **Post-call analysis**. Analyze call recordings to generate conversation transcripts, call summaries, sentiment assessments, and more.
67-
68-
* **Conversation analysis**. Generate transcriptions, summaries, and sentiment assessments from conversation audio recordings.
69-
70-
Start with a template or create a custom analyzer to meet your specific business needs.
79+
## Content Understanding prebuilt audio analyzers
80+
81+
The prebuilt analyzers allow extracting valuable insights into audio content without the need to create an analyzer setup.
82+
83+
All audio analyzers generate transcripts in standard WEBVTT format separated by speaker.
84+
85+
> [!NOTE]
86+
>
87+
> Prebuilt analyzers are set to use multilingual transcription and `returnDetails` enabled.
88+
89+
Content Understanding offers the following prebuilt analyzers:
90+
91+
**Post-call analysis (prebuilt-callCenter)**. Analyze call recordings to generate:
92+
93+
* conversation transcripts with speaker role detection result
94+
* call summary
95+
* call sentiment
96+
* top five articles mentioned
97+
* list of companies mentioned
98+
* list of people (name and title/role) mentioned
99+
* list of relevant call categories
100+
101+
**Example result:**
102+
```json
103+
{
104+
"id": "bc36da27-004f-475e-b808-8b8aead3b566",
105+
"status": "Succeeded",
106+
"result": {
107+
"analyzerId": "prebuilt-callCenter",
108+
"apiVersion": "2025-05-01-preview",
109+
"createdAt": "2025-05-06T22:53:28Z",
110+
"stringEncoding": "utf8",
111+
"warnings": [],
112+
"contents": [
113+
{
114+
"markdown": "# Audio: 00:00.000 => 00:32.183\n\nTranscript\n```\nWEBVTT\n\n00:00.080 --> 00:00.640\n<v Agent>Good day.\n\n00:00.960 --> 00:02.240\n<v Agent>Welcome to Contoso.\n\n00:02.560 --> 00:03.760\n<v Agent>My name is John Doe.\n\n00:03.920 --> 00:05.120\n<v Agent>How can I help you today?\n\n00:05.440 --> 00:06.320\n<v Agent>Yes, good day.\n\n00:06.720 --> 00:08.160\n<v Agent>My name is Maria Smith.\n\n00:08.560 --> 00:11.280\n<v Agent>I would like to inquire about my current point balance.\n\n00:11.680 --> 00:12.560\n<v Agent>No problem.\n\n00:12.880 --> 00:13.920\n<v Agent>I am happy to help.\n\n00:14.240 --> 00:16.720\n<v Agent>I need your date of birth to confirm your identity.\n\n00:17.120 --> 00:19.600\n<v Agent>It is April 19th, 1988.\n\n00:20.000 --> 00:20.480\n<v Agent>Great.\n\n00:20.800 --> 00:24.160\n<v Agent>Your current point balance is 599 points.\n\n00:24.560 --> 00:26.160\n<v Agent>Do you need any more information?\n\n00:26.480 --> 00:27.200\n<v Agent>No, thank you.\n\n00:27.600 --> 00:28.320\n<v Agent>That was all.\n\n00:28.720 --> 00:29.280\n<v Agent>Goodbye.\n\n00:29.680 --> 00:30.320\n<v Agent>You're welcome.\n\n00:30.640 --> 00:31.840\n<v Agent>Goodbye at Contoso.\n```",
115+
"fields": {
116+
"Summary": {
117+
"type": "string",
118+
"valueString": "Maria Smith contacted Contoso to inquire about her current point balance. After confirming her identity with her date of birth, the agent, John Doe, informed her that her balance was 599 points. Maria did not require any further assistance, and the call concluded politely."
119+
},
120+
"Topics": {
121+
"type": "array",
122+
"valueArray": [
123+
{
124+
"type": "string",
125+
"valueString": "Point balance inquiry"
126+
},
127+
{
128+
"type": "string",
129+
"valueString": "Identity confirmation"
130+
},
131+
{
132+
"type": "string",
133+
"valueString": "Customer service"
134+
}
135+
]
136+
},
137+
"Companies": {
138+
"type": "array",
139+
"valueArray": [
140+
{
141+
"type": "string",
142+
"valueString": "Contoso"
143+
}
144+
]
145+
},
146+
"People": {
147+
"type": "array",
148+
"valueArray": [
149+
{
150+
"type": "object",
151+
"valueObject": {
152+
"Name": {
153+
"type": "string",
154+
"valueString": "John Doe"
155+
},
156+
"Role": {
157+
"type": "string",
158+
"valueString": "Agent"
159+
}
160+
}
161+
},
162+
{
163+
"type": "object",
164+
"valueObject": {
165+
"Name": {
166+
"type": "string",
167+
"valueString": "Maria Smith"
168+
},
169+
"Role": {
170+
"type": "string",
171+
"valueString": "Customer"
172+
}
173+
}
174+
}
175+
]
176+
},
177+
"Sentiment": {
178+
"type": "string",
179+
"valueString": "Positive"
180+
},
181+
"Categories": {
182+
"type": "array",
183+
"valueArray": [
184+
{
185+
"type": "string",
186+
"valueString": "Business"
187+
}
188+
]
189+
}
190+
},
191+
"kind": "audioVisual",
192+
"startTimeMs": 0,
193+
"endTimeMs": 32183,
194+
"transcriptPhrases": [
195+
{
196+
"speaker": "Agent",
197+
"startTimeMs": 80,
198+
"endTimeMs": 640,
199+
"text": "Good day.",
200+
"words": []
201+
}, ...
202+
{
203+
"speaker": "Customer",
204+
"startTimeMs": 5440,
205+
"endTimeMs": 6320,
206+
"text": "Yes, good day.",
207+
"words": []
208+
}, ...
209+
]
210+
}
211+
]
212+
}
213+
}
214+
```
215+
216+
**Conversation analysis (prebuilt-audioAnalyzer)**. Analyze recordings to generate:
217+
- conversation transcripts
218+
- conversation summary
219+
220+
**Example result:**
221+
```json
222+
{
223+
"id": "9624cc49-b6b3-4ce5-be6c-e895d8c2484d",
224+
"status": "Succeeded",
225+
"result": {
226+
"analyzerId": "prebuilt-audioAnalyzer",
227+
"apiVersion": "2025-05-01-preview",
228+
"createdAt": "2025-05-06T23:00:12Z",
229+
"stringEncoding": "utf8",
230+
"warnings": [],
231+
"contents": [
232+
{
233+
"markdown": "# Audio: 00:00.000 => 00:32.183\n\nTranscript\n```\nWEBVTT\n\n00:00.080 --> 00:00.640\n<v Speaker 1>Good day.\n\n00:00.960 --> 00:02.240\n<v Speaker 1>Welcome to Contoso.\n\n00:02.560 --> 00:03.760\n<v Speaker 1>My name is John Doe.\n\n00:03.920 --> 00:05.120\n<v Speaker 1>How can I help you today?\n\n00:05.440 --> 00:06.320\n<v Speaker 1>Yes, good day.\n\n00:06.720 --> 00:08.160\n<v Speaker 1>My name is Maria Smith.\n\n00:08.560 --> 00:11.280\n<v Speaker 1>I would like to inquire about my current point balance.\n\n00:11.680 --> 00:12.560\n<v Speaker 1>No problem.\n\n00:12.880 --> 00:13.920\n<v Speaker 1>I am happy to help.\n\n00:14.240 --> 00:16.720\n<v Speaker 1>I need your date of birth to confirm your identity.\n\n00:17.120 --> 00:19.600\n<v Speaker 1>It is April 19th, 1988.\n\n00:20.000 --> 00:20.480\n<v Speaker 1>Great.\n\n00:20.800 --> 00:24.160\n<v Speaker 1>Your current point balance is 599 points.\n\n00:24.560 --> 00:26.160\n<v Speaker 1>Do you need any more information?\n\n00:26.480 --> 00:27.200\n<v Speaker 1>No, thank you.\n\n00:27.600 --> 00:28.320\n<v Speaker 1>That was all.\n\n00:28.720 --> 00:29.280\n<v Speaker 1>Goodbye.\n\n00:29.680 --> 00:30.320\n<v Speaker 1>You're welcome.\n\n00:30.640 --> 00:31.840\n<v Speaker 1>Goodbye at Contoso.\n```",
234+
"fields": {
235+
"Summary": {
236+
"type": "string",
237+
"valueString": "Maria Smith contacted Contoso to inquire about her current point balance. John Doe assisted her by confirming her identity using her date of birth and informed her that her balance was 599 points. Maria expressed no further inquiries, and the conversation concluded politely."
238+
}
239+
},
240+
"kind": "audioVisual",
241+
"startTimeMs": 0,
242+
"endTimeMs": 32183,
243+
"transcriptPhrases": [
244+
{
245+
"speaker": "Speaker 1",
246+
"startTimeMs": 80,
247+
"endTimeMs": 640,
248+
"text": "Good day.",
249+
"words": []
250+
}, ...
251+
{
252+
"speaker": "Speaker 2",
253+
"startTimeMs": 5440,
254+
"endTimeMs": 6320,
255+
"text": "Yes, good day.",
256+
"words": []
257+
}, ...
258+
]
259+
}
260+
]
261+
}
262+
}
263+
```
264+
265+
You can also customize prebuilt analyzers for more fine-grained control of the output by defining custom fields. Customization allows you to use the full power of generative models to extract deep insights from the audio. For example, customization allows you to:
266+
267+
* Generate other insights.
268+
* Control the language of the field extraction output.
269+
* Configure the transcription behavior.
71270

72271
## Input requirements
73272
For a detailed list of supported audio formats, refer to our [Service limits and codecs](../service-limits.md) page.

‎articles/ai-services/content-understanding/concepts/accuracy-confidence.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ms.author: admaheshwari
77
manager: nitinme
88
ms.service: azure-ai-content-understanding
99
ms.topic: reference
10-
ms.date: 04/09/2025
10+
ms.date: 05/19/2025
1111
---
1212

1313
# Interpret and improve confidence and accuracy scores

‎articles/ai-services/content-understanding/concepts/analyzer-templates.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ms.author: kabrow
77
manager: nitinme
88
ms.service: azure-ai-content-understanding
99
ms.topic: overview
10-
ms.date: 05/01/2025
10+
ms.date: 05/19/2025
1111
ms.custom: ignite-2024-understanding-release
1212
---
1313

‎articles/ai-services/content-understanding/concepts/best-practices.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ms.author: jfilcik
77
manager: nitinme
88
ms.service: azure-ai-content-understanding
99
ms.topic: overview
10-
ms.date: 02/24/2025
10+
ms.date: 05/19/2025
1111
---
1212

1313
# Best practices for Content Understanding

‎articles/ai-services/content-understanding/concepts/capabilities.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ms.author: lajanuar
77
manager: nitinme
88
ms.service: azure-ai-content-understanding
99
ms.topic: overview
10-
ms.date: 02/25/2025
10+
ms.date: 05/19/2025
1111
ms.custom: 2025-understanding-release
1212
---
1313

‎articles/ai-services/content-understanding/concepts/retrieval-augmented-generation.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ms.author: tonyeiyalla
77
manager: nitinme
88
ms.service: azure-ai-content-understanding
99
ms.topic: overview
10-
ms.date: 03/16/2025
10+
ms.date: 05/19/2025
1111
ms.custom: 2025-understanding-release
1212
---
1313
# Multimodal retrieval-augmented generation with Content Understanding

‎articles/ai-services/content-understanding/document/overview.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ms.author: lajanuar
77
manager: nitinme
88
ms.service: azure-ai-content-understanding
99
ms.topic: overview
10-
ms.date: 05/01/2025
10+
ms.date: 05/19/2025
1111
ms.custom: ignite-2024-understanding-release
1212
---
1313

‎articles/ai-services/content-understanding/faq.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ metadata:
66
manager: nitinme
77
ms.service: azure-ai-content-understanding
88
ms.topic: faq
9-
ms.date: 04/14/2025
9+
ms.date: 05/19/2025
1010
ms.author: lajanuar
1111
title: Frequently asked questions
1212
summary: |

‎articles/ai-services/content-understanding/glossary.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ author: laujan
66
manager: nitinme
77
ms.service: azure-ai-content-understanding
88
ms.topic: conceptual
9-
ms.date: 04/14/2025
9+
ms.date: 05/19/2025
1010
ms.author: lajanuar
1111
---
1212

‎articles/ai-services/content-understanding/how-to/create-multi-service-resource.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ author: laujan
66
manager: nitinme
77
ms.service: azure-ai-content-understanding
88
ms.topic: how-to
9-
ms.date: 02/19/2025
9+
ms.date: 05/19/2025
1010
ms.custom: ignite-2024-understanding-release, references_regions
1111
ms.author: lajanuar
1212
---

0 commit comments

Comments
 (0)