You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Azure Communication Services provides support for developers to get real-time access to media streamsto capture, analyze and process audio or video content during active calls.
19
+
Azure Communication Services provides support for developers to get real-time access to media streams. You can use real-time access to capture, analyze, and process audio or video content during active calls.
20
20
21
-
Consumption of live audio and video content is very prevalent in our world today in the forms of online meetings, conferences, live events, online classes and customer support. The modern communications world allows people around the globe to connect with anyone anywhere any moment on any matter virtually. With raw media access, developers can analyze audio or video streams for each participant in a call in real-time. In contact centers these streams can be used to run custom AI models for analysis such as your homegrown NLP for conversation analysis or provide real-time insights and suggestions to boost agent productivity. In virtual appointments media streams can be used to analyze sentiment when providing virtual care for patients or provide remote assistance during video calls leveraging Mixed Reality capabilities. This also opens a path for developers to leverage newer innovations with endless possibilities to enhance interaction experiences.
21
+
Consumption of live audio and video content is prevalent in our world today in the forms of online meetings, conferences, live events, online classes and customer support. The modern communications world allows people around the globe to connect with anyone anywhere any moment on any matter virtually. With raw media access, developers can analyze audio or video streams for each participant in a call in real-time.
22
+
23
+
In contact centers, developers can use these streams to run custom AI models for analysis such as your homegrown natural language processing (NLP) for conversation analysis or provide real-time insights and suggestions to boost agent productivity. In virtual appointments, media streams can be used to analyze sentiment when providing virtual care for patients or provide remote assistance during video calls using Mixed Reality capabilities. This ability also opens a path for developers to apply newer innovations with endless possibilities to enhance interaction experiences.
22
24
23
25
The Azure Communication Services SDKs provides access to the media streams from the client and server side to enable developers building more inclusive and richer virtual experiences during voice or video interactions.
24
26
25
27
:::image type="content" source="../media/raw-media/raw-media-overview-1.png" alt-text="diagram of raw media overview use cases.":::
26
28
27
-
## The workflow can be split into three operations:
28
-
• Capture Media: Media can be captured locally via the client SDKs or on the server side.
29
+
## The workflow can be split into three operations
30
+
31
+
- Capture Media: Media can be captured locally via the client SDKs or on the server side.
29
32
30
-
• Process/Transform: Media can be transformed locally on the client (for example add background blur) or be used for processing in a cloud service (for example to use it with your customer NLPU for conversation insights).
33
+
-Process/Transform: Media can be transformed locally on the client (for example add background blur) or be used for processing in a cloud service (for example to use it with your customer NLP for conversation insights).
31
34
32
-
• Provide context or inject back the Transformed Media: The output of the transformed media streams (ex, sentiment analysis) can be used to provide context or augmented media streamscan be injected into the interaction through the client SDK or through the media streaming API via the server SDK.
35
+
-Provide context or inject back the Transformed Media: The output of the transformed media streams, such as sentiment analysis. Use the output to provide context or augmented media streams. You can inject media streams back into the interaction through the client SDK or media streaming API via the server SDK.
33
36
34
37
## Media access via the Calling Client SDK
35
-
During a call, developers can access the audio and video media streams. Outgoing local audio and video media streams can be pre-processed, before being sent to the encoder. Incoming remote captured media streams can be post-processed before playback on screen or speaker. For incoming audio mixed media access, the client calling SDK can have access to the mixed incoming remote audio stream which includes the mixed audio streams of the top four most dominant speakers on the call. For incoming remote unmixed audio the client calling SDK will have access to the individual audio streams of each participant on the call.
36
38
37
-
:::image type="content" source="../media/raw-media/raw-media-overview-2.png" alt-text="diagram of raw media overview architecture.":::
39
+
During a call, developers can access the audio and video media streams. Outgoing local audio and video media streams can be preprocessed, before being sent to the encoder. Incoming remote captured media streams can be post-processed before playback on screen or speaker. For incoming audio mixed media access, the client calling SDK can have access to the mixed incoming remote audio stream, which includes the mixed audio streams of the top four most dominant speakers on the call. For incoming remote unmixed audio, the client calling SDK has access to the individual audio streams of each participant on the call.
38
40
41
+
:::image type="content" source="../media/raw-media/raw-media-overview-2.png" alt-text="diagram of raw media overview architecture.":::
39
42
40
43
## Media access use cases
41
-
• Screen share: Local outgoing video access can be used to enable screen sharing, developers are able to implement the foreground services to capture the frames and send them to be published using the calling SDK OutgoingVirtualVideoStreamOptions.
42
44
43
-
• Background blur: Local outgoing video access can be used to capture the video frames from the camera and implement background blur before sending the blurred frames to be published using the calling SDK OutgoingVirtualVideoStreamOptions.
45
+
- Screen share: Local outgoing video access can be used to enable screen sharing. Developers can implement the foreground services to capture the frames and send them to be published using the calling SDK `OutgoingVirtualVideoStreamOptions`.
44
46
45
-
• Video filters: Local outgoing video access can be used to capture the video frames from the camera and implement AI video filters on the captured frames before sending the video frames to be published using the calling SDK OutgoingVirtualVideoStreamOptions.
47
+
- Background blur: Local outgoing video access can be used to capture the video frames from the camera and implement background blur before sending the blurred frames to be published using the calling SDK `OutgoingVirtualVideoStreamOptions`.
46
48
47
-
• Augmented reality/Virtual reality: Remote incoming video media streams can be captured and augmented with a virtual environment before rendering on the screen.
49
+
- Video filters: Local outgoing video access can be used to capture the video frames from the camera and implement AI video filters on the captured frames before sending the video frames to be published using the calling SDK `OutgoingVirtualVideoStreamOptions`.
48
50
49
-
• Spatial audio: Remote incoming audio access can be used to inject spatial audio into the incoming audio stream.
51
+
- Augmented reality/Virtual reality: Remote incoming video media streams can be captured and augmented with a virtual environment before rendering on the screen.
50
52
53
+
- Spatial audio: Remote incoming audio access can be used to inject spatial audio into the incoming audio stream.
51
54
52
55
:::image type="content" source="../media/raw-media/raw-media-overview-3.png" alt-text="diagram of raw media overview client interfaces.":::
53
56
54
-
55
57
## Next steps
56
58
57
59
> [!div class="nextstepaction"]
58
60
> [Get started with raw media](../../quickstarts/voice-video-calling/get-started-raw-media-access.md)
59
61
60
-
For more information, see the following articles:
62
+
## Related articles
63
+
61
64
- Familiarize yourself with general [call flows](../call-flows.md)
62
65
- Learn about [call types](../voice-video-calling/about-call-types.md)
63
66
-[Plan your PSTN solution](../telephony/plan-solution.md)
titleSuffix: An Azure Communication Services Calling concept doc
3
+
titleSuffix: An Azure Communication Services Calling article
4
4
description: Use Azure Communication Services Calling to review Music Mode.
5
5
author: garchiro7
6
6
ms.author: jorgegarc
7
7
ms.service: azure-communication-services
8
8
ms.subservice: calling
9
9
ms.topic: how-to
10
-
ms.date: 03/01/2024
10
+
ms.date: 06/24/2025
11
11
ms.custom: template-how-to
12
12
---
13
13
14
14
# Music Mode
15
15
16
-
The **music mode** enhances the audio quality for music playback and performance within virtual environments, ensuring clarity and depth in sound reproduction; currently supports a 32-kHz sampling rate at 128 kbps when network bandwidth allows; when network bandwidth is insufficient, the bitrate can be reduced to as low as 48 kbps.
16
+
The **music mode** enhances the audio quality for music playback and performance within virtual environments. Use music mode to ensure clarity and depth in sound reproduction. Music mode currently supports a 32-kHz sampling rate at 128 kbps when network bandwidth allows. If network bandwidth is insufficient, you can reduce the bitrate to as low as 48 kbps.
17
17
18
-
This feature is designed to elevate the audio quality for calls, ensuring the audio is crispy and offering a richer and more immersive audio experience. Also, it reduces audio compression to maintain the original sound, making it ideal for applications ranging from live musical performances and remote music education or music sessions.
18
+
Music mode elevates the audio quality for calls, ensuring the audio is crisp and offers a richer and more immersive audio experience. Music mode also reduces audio compression to maintain the original sound. This efficiency makes it ideal for applications ranging from live musical performances and remote music education or music sessions.
19
19
20
20
Once music mode is enabled, you should consider:
21
21
22
-
- Input and output audio devices that allow high bitrate and sampling rate (two channels, 32 kHz or higher)
23
-
- Enable Control Noise suppression
22
+
- Input and output audio devices that accept high bitrate and sampling rate. Such as two (2) channels at 32 kHz or higher.
23
+
- Enable Control Noise suppression.
24
24
25
25
We recommend using high-quality external loudspeakers, professional microphones, or headsets instead of Bluetooth devices to optimize the music mode.
26
26
27
27
**Note:** music mode only works in 1:1 calls on native platforms and group calls. Currently, music mode doesn't work in 1:1 calls between native and web. By default, music mode is disabled.
28
28
29
-
The Calling native SDK provides an additional set of audio filters that bring a richer experience during the call:
29
+
The Calling native SDK provides another set of audio filters that bring a richer experience during the call:
30
30
31
31
- Echo cancellation. *You can only toggle echo cancellation only if music mode is enabled*
32
32
- Noise suppression. *The currently available modes are `Off`, `Auto`, `Low`, and `High`*
33
33
- Analog Automatic gain control
34
34
- Digital Automatic gain control
35
35
36
36
## Next steps
37
+
37
38
-[Learn how to setup audio filters](../../tutorials/audio-quality-enhancements/add-noise-supression.md)
Copy file name to clipboardExpand all lines: articles/communication-services/quickstarts/voice-video-calling/includes/raw-media/raw-media-access-android.md
+22-21Lines changed: 22 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,25 +1,26 @@
1
1
---
2
-
title: Quickstart - Add raw media access to your app (Android)
3
-
titleSuffix: An Azure Communication Services quickstart
4
-
description: In this quickstart, you learn how to add raw media access calling capabilities to your app by using Azure Communication Services.
2
+
title: Add raw media access to your app (Android)
3
+
titleSuffix: An Azure Communication Services article
4
+
description: This article describes how to add raw media access calling capabilities to your app by using Azure Communication Services.
5
5
author: yassirbisteni
6
6
7
7
ms.author: yassirb
8
-
ms.date: 06/09/2022
8
+
ms.date: 06/24/2025
9
9
ms.topic: quickstart
10
10
ms.service: azure-communication-services
11
11
ms.subservice: calling
12
12
ms.custom: mode-other
13
13
---
14
14
15
-
In this quickstart, you learn how to implement raw media access by using the Azure Communication Services Calling SDK for Android.
15
+
This article describes how to implement raw media access by using the Azure Communication Services Calling SDK for Android.
16
16
17
-
The Azure Communication Services Calling SDK offers APIs that allow apps to generate their own video frames to send to remote participants in a call.
17
+
The Azure Communication Services Calling SDK enables apps to generate their own video frames to send to remote participants in a call.
18
18
19
-
This quickstart builds on [Quickstart: Add 1:1 video calling to your app](../../get-started-with-video-calling.md?pivots=platform-android) for Android.
19
+
This article builds on [Add 1:1 video calling to your app](../../get-started-with-video-calling.md?pivots=platform-android) for Android.
20
20
21
-
## RawAudio access
22
-
Accessing raw audio media gives you access to the incoming audio stream of the call, along with the ability to view and send custom outgoing audio streams during a call.
21
+
## RawAudio access
22
+
23
+
Processing raw audio media gives you access to the incoming audio stream of the call, along with the ability to view and send custom outgoing audio streams during a call.
It's also important to remember to stop the audio stream in the current call`Call` instance:
159
+
Remember to stop the audio stream in the current `Call` instance:
159
160
160
161
```java
161
162
@@ -164,14 +165,14 @@ It's also important to remember to stop the audio stream in the current call `Ca
164
165
165
166
## RawVideo access
166
167
167
-
Because the app generates the video frames, the app must inform the Azure Communication Services Calling SDK about the video formats that the app can generate. This information allows the Azure Communication Services Calling SDK to pick the best video format configuration for the network conditions at that time.
168
+
Because the app generates the video frames, the app must inform the Azure Communication Services Calling SDK about the video formats that the app can generate. This information enables the Azure Communication Services Calling SDK to pick the best video format configuration for the network conditions at that time.
168
169
169
-
## Virtual Video
170
+
## Virtual video
170
171
171
172
### Supported video resolutions
172
173
173
174
| Aspect ratio | Resolution | Maximum FPS |
174
-
|:--:|:-:|:-:|
175
+
|---|---|---|
175
176
| 16x9 | 1080p | 30 |
176
177
| 16x9 | 720p | 30 |
177
178
| 16x9 | 540p | 30 |
@@ -222,7 +223,7 @@ Because the app generates the video frames, the app must inform the Azure Commun
222
223
});
223
224
```
224
225
225
-
6. Create an instance of the following helper classto generate random `RawVideoFrame`'s using `VideoStreamPixelFormat.RGBA`
226
+
6. Create an instance of the following helper classto generate random `RawVideoFrame` using `VideoStreamPixelFormat.RGBA`.
226
227
227
228
```java
228
229
publicclassVideoFrameSender
@@ -377,7 +378,7 @@ Because the Windows system generates the frames, you must implement your own for
377
378
### Supported video resolutions
378
379
379
380
|Aspect ratio |Resolution|MaximumFPS|
380
-
|:--:|:-:|:-:|
381
+
|---|---|---|
381
382
|Anything|Anything up to 1080p |30|
382
383
383
384
### Steps to create a screen share video stream
@@ -412,7 +413,7 @@ Because the Windows system generates the frames, you must implement your own for
412
413
ScreenShareOutgoingVideoStream rawOutgoingVideoStream = new ScreenShareOutgoingVideoStream(rawOutgoingVideoStreamOptions);
413
414
```
414
415
415
-
4. Capture and send the video frame in the following way.
416
+
4. Capture and send the video frame as follows:
416
417
417
418
```java
418
419
private void SendRawVideoFrame()
@@ -437,11 +438,11 @@ Because the Windows system generates the frames, you must implement your own for
437
438
}
438
439
```
439
440
440
-
## Raw Incoming Video
441
+
## Raw incoming video
441
442
442
-
This feature gives you access the video frames inside the `IncomingVideoStream` objects in order to manipulate those frames locally
443
+
This feature gives you access the video frames inside the `IncomingVideoStream` objects to manipulate those frames locally
443
444
444
-
1. Create an instance of `IncomingVideoOptions` that sets through `JoinCallOptions` setting `VideoStreamKind.RawIncoming`
445
+
1. Create an instance of `IncomingVideoOptions` that sets through `JoinCallOptions` setting `VideoStreamKind.RawIncoming`.
445
446
446
447
```java
447
448
IncomingVideoOptions incomingVideoOptions = new IncomingVideoOptions()
@@ -519,6 +520,6 @@ This feature gives you access the video frames inside the `IncomingVideoStream`
0 commit comments