Skip to content

Commit 633ca21

Browse files
authored
Update and rename get-started-with-raw-media-access.md to get-started-raw-media-access.md
1 parent b46e672 commit 633ca21

File tree

2 files changed

+171
-171
lines changed

2 files changed

+171
-171
lines changed
Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
---
2+
title: Quickstart - Add RAW media access to your app (Android)
3+
titleSuffix: An Azure Communication Services quickstart
4+
description: In this quickstart, you'll learn how to add raw media access calling capabilities to your app using Azure Communication Services.
5+
author: LaithRodan
6+
7+
ms.author: larodan
8+
ms.date: 11/18/2021
9+
ms.topic: quickstart
10+
ms.service: azure-communication-services
11+
ms.subservice: calling
12+
13+
14+
---
15+
16+
# Raw media access
17+
18+
[!INCLUDE [Public Preview](../../includes/public-preview-include-document.md)]
19+
20+
## Outbound virtual video device
21+
22+
The ACS Calling SDK offers APIs allowing apps to generate their own video frames to send to remote participants.
23+
24+
This quick start builds upon [QuickStart: Add 1:1 video calling to your app](./get-started-with-video-calling.md?pivots=platform-android) for Android.
25+
26+
27+
## Overview
28+
29+
Once an outbound virtual video device is created, use DeviceManager to make a new virtual video device that behaves just like any other webcam connected to your computer or mobile phone.
30+
31+
Since the app will be generating the video frames, the app must inform the ACS Calling SDK about the video formats the app is capable of generating. This is required to allow the ACS Calling SDK to pick the best video format configuration given the network conditions at any giving time.
32+
33+
The app must register a delegate to get notified about when it should start or stop producing video frames. The delegate event will inform the app which video format is more appropriate for the current network conditions.
34+
35+
The following is an overview of the steps required to create an outbound virtual video device.
36+
37+
1. Create a `VirtualDeviceIdentification` with basic identification information for the new outbound virtual video device.
38+
39+
```java
40+
VirtualDeviceIdentification deviceId = new VirtualDeviceIdentification();
41+
deviceId.setId("QuickStartVirtualVideoDevice");
42+
deviceId.setName("My First Virtual Video Device");
43+
```
44+
45+
2. Create an array of `VideoFormat` with the video formats supported by the app. It is fine to have only one video format supported, but at least one of the provided video formats must be of the `MediaFrameKind::VideoSoftware` type. When multiple formats are provided, the order of the format in the list does not influence or prioritize which one will be used. The selected format is based on external factors like network bandwidth.
46+
47+
```java
48+
ArrayList<VideoFormat> videoFormats = new ArrayList<VideoFormat>();
49+
50+
VideoFormat format = new VideoFormat();
51+
format.setWidth(1280);
52+
format.setHeight(720);
53+
format.setPixelFormat(PixelFormat.RGBA);
54+
format.setMediaFrameKind(MediaFrameKind.VIDEO_SOFTWARE);
55+
format.setFramesPerSecond(30);
56+
format.setStride1(1280 * 4); // It is times 4 because RGBA is a 32-bit format.
57+
58+
videoFormats.add(format);
59+
```
60+
61+
3. Create `OutboundVirtualVideoDeviceOptions` and set `DeviceIdentification` and `VideoFormats` with the previously created objects.
62+
63+
```java
64+
OutboundVirtualVideoDeviceOptions m_options = new OutboundVirtualVideoDeviceOptions();
65+
66+
// ...
67+
68+
m_options.setDeviceIdentification(deviceId);
69+
m_options.setVideoFormats(videoFormats);
70+
```
71+
72+
4. Make sure the `OutboundVirtualVideoDeviceOptions::OnFlowChanged` delegate is defined. This delegate will inform its listener about events requiring the app to start or stop producing video frames. In this quick start, `m_mediaFrameSender` is used as trigger to let the app know when it's time to start generating frames. Feel free to use any mechanism in your app as a trigger.
73+
74+
```java
75+
private MediaFrameSender m_mediaFrameSender;
76+
77+
// ...
78+
79+
m_options.addOnFlowChangedListener(virtualDeviceFlowControlArgs -> {
80+
if (virtualDeviceFlowControlArgs.getMediaFrameSender().getRunningState() == VirtualDeviceRunningState.STARTED) {
81+
// Tell the app's frame generator to start producing frames.
82+
m_mediaFrameSender = virtualDeviceFlowControlArgs.getMediaFrameSender();
83+
} else {
84+
// Tell the app's frame generator to stop producing frames.
85+
m_mediaFrameSender = null;
86+
}
87+
});
88+
```
89+
90+
5. Use `DeviceManager::CreateOutboundVirtualVideoDevice` to create an outbound virtual video device. The returning `OutboundVirtualVideoDevice` should be kept alive as long as the app needs to keep acting as a virtual video device. It is ok to register multiple outbound virtual video devices per app.
91+
92+
```java
93+
private OutboundVirtualVideoDevice m_outboundVirtualVideoDevice;
94+
95+
// ...
96+
97+
m_outboundVirtualVideoDevice = m_deviceManager.createOutboundVirtualVideoDevice(m_options).get();
98+
```
99+
100+
6. Tell device manager to use the recently created virtual camera on calls.
101+
102+
```java
103+
private LocalVideoStream m_localVideoStream;
104+
105+
// ...
106+
107+
for (VideoDeviceInfo videoDeviceInfo : m_deviceManager.getCameras())
108+
{
109+
String deviceId = videoDeviceInfo.getId();
110+
if (deviceId.equalsIgnoreCase("QuickStartVirtualVideoDevice")) // Same id used in step 1.
111+
{
112+
m_localVideoStream = LocalVideoStream(videoDeviceInfo, getApplicationContext());
113+
}
114+
}
115+
```
116+
117+
7. In a non-UI thread or loop in the app, cast the `MediaFrameSender` to the appropriate type defined by the `MediaFrameKind` property of `VideoFormat`. For example, cast it to `SoftwareBasedVideoFrame` and then call the `send` method according to the number of planes defined by the MediaFormat.
118+
After that, create the ByteBuffer backing the video frame if needed. Then, update the content of the video frame. Finally, send the video frame to other participants with the `sendFrame` API.
119+
120+
```java
121+
java.nio.ByteBuffer plane1 = null;
122+
Random rand = new Random();
123+
byte greyValue = 0;
124+
125+
// ...
126+
java.nio.ByteBuffer plane1 = null;
127+
Random rand = new Random();
128+
129+
while (m_outboundVirtualVideoDevice != null) {
130+
while (m_mediaFrameSender != null) {
131+
if (m_mediaFrameSender.getMediaFrameKind() == MediaFrameKind.VIDEO_SOFTWARE) {
132+
SoftwareBasedVideoFrame sender = (SoftwareBasedVideoFrame) m_mediaFrameSender;
133+
VideoFormat videoFormat = sender.getVideoFormat();
134+
135+
// Gets the timestamp for when the video frame has been created.
136+
// This allows better synchronization with audio.
137+
int timeStamp = sender.getTimestamp();
138+
139+
// Adjusts frame dimensions to the video format that network conditions can manage.
140+
if (plane1 == null || videoFormat.getStride1() * videoFormat.getHeight() != plane1.capacity()) {
141+
plane1 = ByteBuffer.allocateDirect(videoFormat.getStride1() * videoFormat.getHeight());
142+
plane1.order(ByteOrder.nativeOrder());
143+
}
144+
145+
// Generates random gray scaled bands as video frame.
146+
int bandsCount = rand.nextInt(15) + 1;
147+
int bandBegin = 0;
148+
int bandThickness = videoFormat.getHeight() * videoFormat.getStride1() / bandsCount;
149+
150+
for (int i = 0; i < bandsCount; ++i) {
151+
byte greyValue = (byte)rand.nextInt(254);
152+
java.util.Arrays.fill(plane1.array(), bandBegin, bandBegin + bandThickness, greyValue);
153+
bandBegin += bandThickness;
154+
}
155+
156+
// Sends video frame to the other participants in the call.
157+
FrameConfirmation fr = sender.sendFrame(plane1, timeStamp).get();
158+
159+
// Waits before generating the next video frame.
160+
// Video format defines how many frames per second app must generate.
161+
Thread.sleep((long) (1000.0f / videoFormat.getFramesPerSecond()));
162+
}
163+
}
164+
165+
// Virtual camera hasn't been created yet.
166+
// Let's wait a little bit before checking again.
167+
// This is for demo only purposes.
168+
// Feel free to use a better synchronization mechanism.
169+
Thread.sleep(100);
170+
}
171+
```

articles/communication-services/quickstarts/voice-video-calling/get-started-with-raw-media-access.md

Lines changed: 0 additions & 171 deletions
This file was deleted.

0 commit comments

Comments
 (0)