Skip to content

Commit f8cae48

Browse files
DanielNovakDanielNovak
authored
Add video filter support (#829)
1 parent 86bed9f commit f8cae48

File tree

15 files changed

+567
-53
lines changed

15 files changed

+567
-53
lines changed

‎docusaurus/docs/Android/06-advanced/05-apply-video-filters.mdx‎

Lines changed: 127 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,132 @@ title: Video & Audio filters
33
description: How to build video or audio filters
44
---
55

6-
## Apply Custom Video Filters
6+
## Video Filters
77

8-
// TODO - cover how to apply custom filters, where they live/exist and some common examples
8+
Some calling apps allow filters to be applied to the current user's video, such as blurring the background, adding AR elements (glasses, moustaches, etc) or applying image filters (such as sepia, bloom etc). StreamVideo's Android SDK has support for injecting your custom filter into the calling experience.
99

10-
// sepia
11-
// grayscale
10+
How does this work? You can inject a filter through `Call.videoFilter`, you will receive each frame of the user's local video as `Bitmap`, allowing you to apply the filters (by mutating the `Bitmap`). This way you have complete freedom over the processing pipeline.
11+
12+
## Adding a Video Filter
13+
14+
Create a `BitmapVideoFilter` or `RawVideoFilter` instance in your project. Here is how the abstract classes look like:
15+
16+
```kotlin
17+
abstract class BitmapVideoFilter : VideoFilter() {
18+
fun filter(bitmap: Bitmap)
19+
}
20+
21+
abstract class RawVideoFilter : VideoFilter() {
22+
abstract fun filter(videoFrame: VideoFrame, surfaceTextureHelper: SurfaceTextureHelper): VideoFrame
23+
}
24+
```
25+
The `BitmapVideoFilter` is a simpler filter that gives you a `Bitmap` which you can then manipulate directly. But it's less performant than using the `RawVideoFilter` which gives you direct access to `VideoFrame` from WebRTC and there is no overhead compared to `BitmapVideoFilter` (like YUV <-> ARGB conversions).
26+
27+
And then set the video filter into `Call.videoFilter`.
28+
29+
We can create and set a simple black and white filter like this:
30+
```kotlin
31+
call.videoFilter = object: BitmapVideoFilter() {
32+
override fun filter(bitmap: Bitmap) {
33+
val c = Canvas(bitmap)
34+
val paint = Paint()
35+
val cm = ColorMatrix()
36+
cm.setSaturation(0f)
37+
val f = ColorMatrixColorFilter(cm)
38+
paint.colorFilter = f
39+
c.drawBitmap(bitmap, 0f, 0f, paint)
40+
}
41+
}
42+
```
43+
44+
:::note
45+
You need to manipulate the original bitmap instance to apply the filters. You can of course create a new bitmap in the process, but you need to then draw it on the `bitmap` instance you get in the `filter` callback
46+
:::
47+
48+
## Adding AI Filters
49+
50+
In some cases, you might also want to apply AI filters. That can be an addition to the user's face (glasses, moustaches, etc), or an ML filter. In this section this use-case will be covered. Specifically, you will use the [Selfie Segmentation](https://developers.google.com/ml-kit/vision/selfie-segmentation/android) from Google's ML kit to change the background behind you.
51+
52+
First include the necessary dependency (check for latest version [here](https://developers.google.com/ml-kit/vision/selfie-segmentation/android#before_you_begin)
53+
54+
```gradle
55+
dependencies {
56+
implementation 'com.google.mlkit:segmentation-selfie:16.0.0-beta4'
57+
}
58+
```
59+
60+
Create a class that will hold your custom filter. The initialisation of the `Segmentation` class is done according to the [official docs](https://developers.google.com/ml-kit/vision/selfie-segmentation/android#enable_raw_size_mask).
61+
62+
```kotlin
63+
class SelfieSegmentation {
64+
65+
private val options =
66+
SelfieSegmenterOptions.Builder()
67+
.setDetectorMode(SelfieSegmenterOptions.STREAM_MODE)
68+
.enableRawSizeMask()
69+
.build()
70+
private val segmenter = Segmentation.getClient(options)
71+
72+
fun applyFilter(bitmap: Bitmap) {
73+
// Send the bitmap into ML Kit for processing
74+
val mlImage = InputImage.fromBitmap(bitmap, 0)
75+
val task = segmenter.process(mlImage)
76+
// Wait for result synchronously on same thread
77+
val mask = Tasks.await(task)
78+
79+
val isRawSizeMaskEnabled = mask.width != bitmap.width || mask.height != bitmap.height
80+
val scaleX = bitmap.width * 1f / mask.width
81+
val scaleY = bitmap.height * 1f / mask.height
82+
83+
// Create a bitmap mask to cover the background
84+
val maskBitmap = Bitmap.createBitmap(
85+
maskColorsFromByteBuffer(mask), mask.width, mask.height, Bitmap.Config.ARGB_8888
86+
)
87+
// Create a canvas from the frame bitmap
88+
val canvas = Canvas(bitmap)
89+
val matrix = Matrix()
90+
if (isRawSizeMaskEnabled) {
91+
matrix.preScale(scaleX, scaleY)
92+
}
93+
// And now draw the bitmap mask onto the original bitmap
94+
canvas.drawBitmap(maskBitmap, matrix, null)
95+
96+
maskBitmap.recycle()
97+
}
98+
99+
private fun maskColorsFromByteBuffer(mask: SegmentationMask): IntArray {
100+
val colors = IntArray(mask.width * mask.height)
101+
for (i in 0 until mask.width * mask.height) {
102+
val backgroundLikelihood = 1 - mask.buffer.float
103+
if (backgroundLikelihood > 0.9) {
104+
colors[i] = Color.argb(128, 0, 0, 255)
105+
} else if (backgroundLikelihood > 0.2) {
106+
val alpha = (182.9 * backgroundLikelihood - 36.6 + 0.5).toInt()
107+
colors[i] = Color.argb(alpha, 0, 0, 255)
108+
}
109+
}
110+
return colors
111+
}
112+
}
113+
```
114+
115+
And now set the custom filter into our SDK:
116+
117+
```kotlin
118+
call.videoFilter = object: BitmapVideoFilter() {
119+
120+
val selfieFilter = SelfieSegmentation()
121+
122+
override fun filter(bitmap: Bitmap) {
123+
selfieFilter.applyFilter(bitmap)
124+
}
125+
}
126+
```
127+
128+
The result:
129+
130+
![Stream Filter](../assets/screenshot_video_filter.png)
131+
132+
## Audio Filters
133+
134+
TODO

‎docusaurus/docs/Android/assets/screenshot_video_filter.png‎

2.53 MB
Loading

‎dogfooding/src/main/kotlin/io/getstream/video/android/ui/call/SettingsMenu.kt‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
package io.getstream.video.android.ui.call
1818

1919
import android.app.Activity
20+
import android.graphics.Bitmap
2021
import android.media.projection.MediaProjectionManager
2122
import android.widget.Toast
2223
import androidx.activity.compose.rememberLauncherForActivityResult
@@ -45,7 +46,9 @@ import androidx.compose.ui.unit.dp
4546
import androidx.compose.ui.window.Popup
4647
import io.getstream.video.android.compose.theme.VideoTheme
4748
import io.getstream.video.android.core.Call
49+
import io.getstream.video.android.core.call.video.BitmapVideoFilter
4850
import io.getstream.video.android.ui.common.R
51+
import io.getstream.video.android.util.SampleVideoFilter
4952
import kotlinx.coroutines.launch
5053

5154
@Composable
@@ -144,6 +147,34 @@ internal fun SettingsMenu(
144147

145148
Spacer(modifier = Modifier.height(12.dp))
146149

150+
Row(
151+
modifier = Modifier.clickable {
152+
if (call.videoFilter == null) {
153+
call.videoFilter = object : BitmapVideoFilter() {
154+
override fun filter(bitmap: Bitmap) {
155+
SampleVideoFilter.toGrayscale(bitmap)
156+
}
157+
}
158+
} else {
159+
call.videoFilter = null
160+
}
161+
},
162+
) {
163+
Icon(
164+
painter = painterResource(id = R.drawable.stream_video_ic_fullscreen_exit),
165+
tint = VideoTheme.colors.textHighEmphasis,
166+
contentDescription = null,
167+
)
168+
169+
Text(
170+
modifier = Modifier.padding(start = 20.dp),
171+
text = "Toggle video filter",
172+
color = VideoTheme.colors.textHighEmphasis,
173+
)
174+
}
175+
176+
Spacer(modifier = Modifier.height(12.dp))
177+
147178
if (showDebugOptions) {
148179
Row(
149180
modifier = Modifier.clickable {

stream-video-android-core/src/main/kotlin/io/getstream/video/android/core/filter/AudioFilter.kt renamed to dogfooding/src/main/kotlin/io/getstream/video/android/util/SampleVideoFilter.kt

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,23 @@
1414
* limitations under the License.
1515
*/
1616

17-
package io.getstream.video.android.core.filter
17+
package io.getstream.video.android.util
1818

19-
public interface AudioFilter
19+
import android.graphics.Bitmap
20+
import android.graphics.Canvas
21+
import android.graphics.ColorMatrix
22+
import android.graphics.ColorMatrixColorFilter
23+
import android.graphics.Paint
24+
25+
object SampleVideoFilter {
26+
27+
fun toGrayscale(bmpOriginal: Bitmap) {
28+
val c = Canvas(bmpOriginal)
29+
val paint = Paint()
30+
val cm = ColorMatrix()
31+
cm.setSaturation(0f)
32+
val f = ColorMatrixColorFilter(cm)
33+
paint.colorFilter = f
34+
c.drawBitmap(bmpOriginal, 0f, 0f, paint)
35+
}
36+
}

‎gradle/libs.versions.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ landscapist = "2.2.8"
3232
accompanist = "0.30.1"
3333
telephoto = "0.3.0"
3434
audioswitch = "1.1.8"
35+
libyuv = "0.28.0"
3536

3637
wire = "4.7.0"
3738
okhttp = "4.11.0"
@@ -120,6 +121,8 @@ telephoto = { group = "me.saket.telephoto", name = "zoomable", version.ref = "te
120121

121122
audioswitch = { group = "com.twilio", name = "audioswitch", version.ref = "audioswitch"}
122123

124+
libyuv = { group = "io.github.crow-misia.libyuv", name = "libyuv-android", version.ref = "libyuv"}
125+
123126
wire-runtime = { group = "com.squareup.wire", name = "wire-runtime", version.ref = "wire" }
124127
retrofit = { group = "com.squareup.retrofit2", name = "retrofit", version.ref = "retrofit" }
125128
retrofit-moshi = { group = "com.squareup.retrofit2", name = "converter-moshi", version.ref = "retrofit" }

‎stream-video-android-core/api/stream-video-android-core.api‎

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ public final class io/getstream/video/android/core/Call {
2121
public final fun getState ()Lio/getstream/video/android/core/CallState;
2222
public final fun getType ()Ljava/lang/String;
2323
public final fun getUser ()Lio/getstream/video/android/model/User;
24+
public final fun getVideoFilter ()Lio/getstream/video/android/core/call/video/VideoFilter;
2425
public final fun goLive (ZZZLkotlin/coroutines/Continuation;)Ljava/lang/Object;
2526
public static synthetic fun goLive$default (Lio/getstream/video/android/core/Call;ZZZLkotlin/coroutines/Continuation;ILjava/lang/Object;)Ljava/lang/Object;
2627
public final fun grantPermissions (Ljava/lang/String;Ljava/util/List;Lkotlin/coroutines/Continuation;)Ljava/lang/Object;
@@ -50,6 +51,7 @@ public final class io/getstream/video/android/core/Call {
5051
public final fun sendReaction (Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Lkotlin/coroutines/Continuation;)Ljava/lang/Object;
5152
public static synthetic fun sendReaction$default (Lio/getstream/video/android/core/Call;Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Lkotlin/coroutines/Continuation;ILjava/lang/Object;)Ljava/lang/Object;
5253
public final fun sendStats (Ljava/util/Map;Lkotlin/coroutines/Continuation;)Ljava/lang/Object;
54+
public final fun setVideoFilter (Lio/getstream/video/android/core/call/video/VideoFilter;)V
5355
public final fun setVisibility (Ljava/lang/String;Lstream/video/sfu/models/TrackType;Z)V
5456
public final fun startHLS (Lkotlin/coroutines/Continuation;)Ljava/lang/Object;
5557
public final fun startRecording (Lkotlin/coroutines/Continuation;)Ljava/lang/Object;
@@ -741,12 +743,10 @@ public final class io/getstream/video/android/core/StreamVideoBuilder {
741743
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;)V
742744
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;)V
743745
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;)V
744-
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;Ljava/util/List;)V
745-
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;Ljava/util/List;Ljava/util/List;)V
746-
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;Ljava/util/List;Ljava/util/List;J)V
747-
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;Ljava/util/List;Ljava/util/List;JZ)V
748-
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;Ljava/util/List;Ljava/util/List;JZLjava/lang/String;)V
749-
public synthetic fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;Ljava/util/List;Ljava/util/List;JZLjava/lang/String;ILkotlin/jvm/internal/DefaultConstructorMarker;)V
746+
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;J)V
747+
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;JZ)V
748+
public fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;JZLjava/lang/String;)V
749+
public synthetic fun <init> (Landroid/content/Context;Ljava/lang/String;Lio/getstream/video/android/core/GEO;Lio/getstream/video/android/model/User;Ljava/lang/String;Lkotlin/jvm/functions/Function2;Lio/getstream/video/android/core/logging/LoggingLevel;Lio/getstream/video/android/core/notifications/NotificationConfig;Lkotlin/jvm/functions/Function1;JZLjava/lang/String;ILkotlin/jvm/internal/DefaultConstructorMarker;)V
750750
public final fun build ()Lio/getstream/video/android/core/StreamVideo;
751751
public final fun getScope ()Lkotlinx/coroutines/CoroutineScope;
752752
}
@@ -958,7 +958,6 @@ public final class io/getstream/video/android/core/call/connection/StreamPeerCon
958958
public final fun makeAudioTrack (Lorg/webrtc/AudioSource;Ljava/lang/String;)Lorg/webrtc/AudioTrack;
959959
public final fun makePeerConnection (Lkotlinx/coroutines/CoroutineScope;Lorg/webrtc/PeerConnection$RTCConfiguration;Lio/getstream/video/android/core/model/StreamPeerType;Lorg/webrtc/MediaConstraints;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function2;Lkotlin/jvm/functions/Function2;I)Lio/getstream/video/android/core/call/connection/StreamPeerConnection;
960960
public static synthetic fun makePeerConnection$default (Lio/getstream/video/android/core/call/connection/StreamPeerConnectionFactory;Lkotlinx/coroutines/CoroutineScope;Lorg/webrtc/PeerConnection$RTCConfiguration;Lio/getstream/video/android/core/model/StreamPeerType;Lorg/webrtc/MediaConstraints;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function2;Lkotlin/jvm/functions/Function2;IILjava/lang/Object;)Lio/getstream/video/android/core/call/connection/StreamPeerConnection;
961-
public final fun makeVideoSource (Z)Lorg/webrtc/VideoSource;
962961
public final fun makeVideoTrack (Lorg/webrtc/VideoSource;Ljava/lang/String;)Lorg/webrtc/VideoTrack;
963962
public final fun setAudioSampleCallback (Lkotlin/jvm/functions/Function1;)V
964963
}
@@ -2740,6 +2739,19 @@ public final class io/getstream/video/android/core/call/stats/model/discriminato
27402739
public final fun fromAlias (Ljava/lang/String;)Lio/getstream/video/android/core/call/stats/model/discriminator/RtcReportType;
27412740
}
27422741

2742+
public abstract class io/getstream/video/android/core/call/video/BitmapVideoFilter : io/getstream/video/android/core/call/video/VideoFilter {
2743+
public fun <init> ()V
2744+
public abstract fun filter (Landroid/graphics/Bitmap;)V
2745+
}
2746+
2747+
public abstract class io/getstream/video/android/core/call/video/RawVideoFilter : io/getstream/video/android/core/call/video/VideoFilter {
2748+
public fun <init> ()V
2749+
public abstract fun filter (Lorg/webrtc/VideoFrame;Lorg/webrtc/SurfaceTextureHelper;)Lorg/webrtc/VideoFrame;
2750+
}
2751+
2752+
public class io/getstream/video/android/core/call/video/VideoFilter {
2753+
}
2754+
27432755
public final class io/getstream/video/android/core/dispatchers/DispatcherProvider {
27442756
public static final field INSTANCE Lio/getstream/video/android/core/dispatchers/DispatcherProvider;
27452757
public final fun getDefault ()Lkotlinx/coroutines/CoroutineDispatcher;
@@ -3081,9 +3093,6 @@ public final class io/getstream/video/android/core/filter/AndFilterObject : io/g
30813093
public fun toString ()Ljava/lang/String;
30823094
}
30833095

3084-
public abstract interface class io/getstream/video/android/core/filter/AudioFilter {
3085-
}
3086-
30873096
public final class io/getstream/video/android/core/filter/AutocompleteFilterObject : io/getstream/video/android/core/filter/FilterObject {
30883097
public final fun component1 ()Ljava/lang/String;
30893098
public final fun component2 ()Ljava/lang/String;
@@ -3295,9 +3304,6 @@ public final class io/getstream/video/android/core/filter/OrFilterObject : io/ge
32953304
public fun toString ()Ljava/lang/String;
32963305
}
32973306

3298-
public abstract interface class io/getstream/video/android/core/filter/VideoFilter {
3299-
}
3300-
33013307
public abstract interface annotation class io/getstream/video/android/core/internal/InternalStreamVideoApi : java/lang/annotation/Annotation {
33023308
}
33033309

‎stream-video-android-core/build.gradle.kts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,9 @@ dependencies {
136136

137137
implementation(libs.audioswitch)
138138

139+
// video filter dependencies
140+
implementation (libs.libyuv)
141+
139142
// androidx
140143
implementation(libs.androidx.core.ktx)
141144
implementation(libs.androidx.lifecycle.runtime)

‎stream-video-android-core/src/main/kotlin/io/getstream/video/android/core/Call.kt‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ import io.getstream.result.Result.Failure
2626
import io.getstream.result.Result.Success
2727
import io.getstream.video.android.core.call.RtcSession
2828
import io.getstream.video.android.core.call.utils.SoundInputProcessor
29+
import io.getstream.video.android.core.call.video.VideoFilter
2930
import io.getstream.video.android.core.events.VideoEventListener
3031
import io.getstream.video.android.core.internal.InternalStreamVideoApi
3132
import io.getstream.video.android.core.model.MuteUsersData
@@ -126,6 +127,11 @@ public class Call(
126127
/** The cid is type:id */
127128
val cid = "$type:$id"
128129

130+
/**
131+
* Set a custom [VideoFilter] that will be applied to the video stream coming from your device.
132+
*/
133+
var videoFilter: VideoFilter? = null
134+
129135
/**
130136
* Called by the [CallHealthMonitor] when the ICE restarts failed after
131137
* several retries. At this point we can do a full reconnect.

0 commit comments

Comments
 (0)