Skip to content

Commit 6fe793f

Browse files
authored
Update media-access.md
1 parent 913cebd commit 6fe793f

File tree

1 file changed

+10
-0
lines changed
  • articles/communication-services/concepts/voice-video-calling

1 file changed

+10
-0
lines changed

articles/communication-services/concepts/voice-video-calling/media-access.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,24 +22,34 @@ Azure Communication Services provides support for developers to get real-time ac
2222
Consumption of live audio and video content is very prevalent in our world today in the forms of online meetings, conferences, live events, online classes and customer support. The modern communications world allows people around the globe to connect with anyone anywhere any moment on any matter virtually. With raw media access, developers can analyze audio or video streams for each participant in a call in real-time. In contact centers these streams can be used to run custom AI models for analysis such as your homegrown NLP for conversation analysis or provide real-time insights and suggestions to boost agent productivity. In virtual appointments media streams can be used to analyze sentiment when providing virtual care for patients or provide remote assistance during video calls leveraging Mixed Reality capabilities. This also opens a path for developers to leverage newer innovations with endless possibilities to enhance interaction experiences.
2323

2424
The Azure Communication Services SDKs provides access to the media streams from the client and server side to enable developers building more inclusive and richer virtual experiences during voice or video interactions.
25+
2526
:::image type="content" source="../media/raw-media/raw-media-overview-1.png" alt-text="diagram of raw media overview use cases.":::
2627

2728
## The workflow can be split into three operations:
2829
• Capture Media: Media can be captured locally via the client SDKs or on the server side.
30+
2931
• Process/Transform: Media can be transformed locally on the client (for example add background blur) or be used for processing in a cloud service (for example to use it with your customer NLPU for conversation insights).
32+
3033
• Provide context or inject back the Transformed Media: The output of the transformed media streams (ex, sentiment analysis) can be used to provide context or augmented media streams can be injected into the interaction through the client SDK or through the media streaming API via the server SDK.
3134

3235
## Media access via the Calling Client SDK
3336
During a call, developers can access the audio and video media streams. Outgoing local audio and video media streams can be pre-processed, before being sent to the encoder. Incoming remote captured media streams can be post-processed before playback on screen or speaker. For incoming audio mixed media access, the client calling SDK can have access to the mixed incoming remote audio stream which includes the mixed audio streams of the top four most dominant speakers on the call. For incoming remote unmixed audio the client calling SDK will have access to the individual audio streams of each participant on the call.
37+
3438
:::image type="content" source="../media/raw-media/raw-media-overview-2.png" alt-text="diagram of raw media overview architecture.":::
3539

40+
3641
## Media access use cases
3742
• Screen share: Local outgoing video access can be used to enable screen sharing, developers are able to implement the foreground services to capture the frames and send them to be published using the calling SDK OutgoingVirtualVideoStreamOptions.
43+
3844
• Background blur: Local outgoing video access can be used to capture the video frames from the camera and implement background blur before sending the blurred frames to be published using the calling SDK OutgoingVirtualVideoStreamOptions.
45+
3946
• Video filters: Local outgoing video access can be used to capture the video frames from the camera and implement AI video filters on the captured frames before sending the video frames to be published using the calling SDK OutgoingVirtualVideoStreamOptions.
47+
4048
• Augmented reality/Virtual reality: Remote incoming video media streams can be captured and augmented with a virtual environment before rendering on the screen.
49+
4150
• Spatial audio: Remote incoming audio access can be used to inject spatial audio into the incoming audio stream.
4251

52+
4353
:::image type="content" source="../media/raw-media/raw-media-overview-3.png" alt-text="diagram of raw media overview client interfaces.":::
4454

4555

0 commit comments

Comments
 (0)