Skip to content

Commit 7a8a367

Browse files
pareenavermapareenaverma
authored
Merge pull request #1434 from hanyin-arm/hanyin-selfie-app
New Learning Path: "Build a Hands-Free Selfie app with Modern Android Development and MediaPipe Multimodal AI"
2 parents cb91bfc + 873183e commit 7a8a367

21 files changed

+1875
-0
lines changed

‎assets/contributors.csv‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,4 @@ Chen Zhang,Zilliz,,,,
4848
Tianyu Li,Arm,,,,
4949
Georgios Mermigkis,VectorCamp,gMerm,georgios-mermigkis,,https://vectorcamp.gr/
5050
Ben Clark,Arm,,,,
51+
Han Yin,Arm,hanyin-arm,nacosiren,,

‎content/learning-paths/smartphones-and-mobile/build-android-selfie-app-using-mediapipe-multimodality/2-app-scaffolding.md‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
---
2+
title: Scaffold a new Android project
3+
weight: 2
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
This learning path will teach you to architect an app following [modern Android architecture](https://developer.android.com/courses/pathways/android-architecture) design with a focus on the [UI layer](https://developer.android.com/topic/architecture/ui-layer).
10+
11+
## Development environment setup
12+
13+
Download and install the latest version of [Android Studio](https://developer.android.com/studio/) on your host machine.
14+
15+
This learning path's instructions and screenshots are taken on macOS with Apple Silicon, but you may choose any of the supported hardware systems as described [here](https://developer.android.com/studio/install).
16+
17+
Upon first installation, open Android Studio and proceed with the default or recommended settings. Accept license agreements and let Android Studio download all the required assets.
18+
19+
Before you proceed to coding, here are some tips that might come handy:
20+
21+
{{% notice Tip %}}
22+
1. To navigate to a file, simply double-tap `Shift` key and input the file name, then select the correct result using `Up` & `Down` arrow keys and then tap `Enter`.
23+
24+
2. Every time after you copy-paste a code block from this learning path, make sure you **import the correct classes** and resolved the errors. Refer to [this doc](https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html) to learn more.
25+
{{% /notice %}}
26+
27+
## Create a new Android project
28+
29+
1. Navigate to **File > New > New Project...**.
30+
31+
2. Select **Empty Views Activity** in **Phone and Tablet** galary as shown below, then click **Next**.
32+
![Empty Views Activity](images/2/empty%20project.png)
33+
34+
3. Proceed with a cool project name and default configurations as shown below. Make sure that **Language** is set to **Kotlin**, and that **Build configuration language** is set to **Kotlin DSL**.
35+
![Project configuration](images/2/project%20config.png)
36+
37+
### Introduce CameraX dependencies
38+
39+
[CameraX](https://developer.android.com/media/camera/camerax) is a Jetpack library, built to help make camera app development easier. It provides a consistent, easy-to-use API that works across the vast majority of Android devices with a great backward-compatibility.
40+
41+
1. Wait for Android Studio to sync project with Gradle files, this make take up to several minutes.
42+
43+
2. Once project is synced, navigate to `libs.versions.toml` in your project's root directory as shown below. This file serves as the version catalog for all dependencies used in the project.
44+
45+
![version catalog](images/2/dependency%20version%20catalog.png)
46+
47+
{{% notice Info %}}
48+
49+
For more information on version catalogs, please refer to [this doc](https://developer.android.com/build/migrate-to-catalogs).
50+
51+
{{% /notice %}}
52+
53+
3. Append the following line to the end of `[versions]` section. This defines the version of CameraX libraries we will be using.
54+
```toml
55+
camerax = "1.4.0"
56+
```
57+
58+
4. Append the following lines to the end of `[libraries]` section. This declares the group, name and version of CameraX dependencies.
59+
60+
```toml
61+
camera-core = { group = "androidx.camera", name = "camera-core", version.ref = "camerax" }
62+
camera-camera2 = { group = "androidx.camera", name = "camera-camera2", version.ref = "camerax" }
63+
camera-lifecycle = { group = "androidx.camera", name = "camera-lifecycle", version.ref = "camerax" }
64+
camera-view = { group = "androidx.camera", name = "camera-view", version.ref = "camerax" }
65+
```
66+
67+
5. Navigate to `build.gradle.kts` in your project's `app` directory, then insert the following lines into `dependencies` block. This introduces the above dependencies into the `app` subproject.
68+
69+
```kotlin
70+
implementation(libs.camera.core)
71+
implementation(libs.camera.camera2)
72+
implementation(libs.camera.lifecycle)
73+
implementation(libs.camera.view)
74+
```
75+
76+
## Enable view binding
77+
78+
1. Within the above `build.gradle.kts` file, append the following lines to the end of `android` block to enable view binding feature.
79+
80+
```kotlin
81+
buildFeatures {
82+
viewBinding = true
83+
}
84+
```
85+
86+
2. You should be seeing a notification shows up, as shown below. Click **"Sync Now"** to sync your project.
87+
88+
![Gradle sync](images/2/gradle%20sync.png)
89+
90+
{{% notice Tip %}}
91+
92+
You may also click the __"Sync Project with Gradle Files"__ button in the toolbar or pressing the corresponding shorcut to start a sync.
93+
94+
![Sync Project with Gradle Files](images/2/sync%20project%20with%20gradle%20files.png)
95+
{{% /notice %}}
96+
97+
3. Navigate to `MainActivity.kt` source file and make following changes. This inflates the layout file into a view binding object and stores it in a member variable within the view controller for easier access later.
98+
99+
![view binding](images/2/view%20binding.png)
100+
101+
## Configure CameraX preview
102+
103+
1. **Replace** the placeholder "Hello World!" `TextView` within the layout file `activity_main.xml` with a camera preview view:
104+
105+
```xml
106+
<androidx.camera.view.PreviewView
107+
android:id="@+id/view_finder"
108+
android:layout_width="match_parent"
109+
android:layout_height="match_parent"
110+
app:scaleType="fillStart" />
111+
```
112+
113+
114+
2. Add the following member variables to `MainActivity.kt` to store camera related objects:
115+
116+
```kotlin
117+
// Camera
118+
private var camera: Camera? = null
119+
private var cameraProvider: ProcessCameraProvider? = null
120+
private var preview: Preview? = null
121+
```
122+
123+
3. Add two new private methods named `setupCamera()` and `bindCameraUseCases()` within `MainActivity.kt`:
124+
125+
```kotlin
126+
private fun setupCamera() {
127+
viewBinding.viewFinder.post {
128+
cameraProvider?.unbindAll()
129+
130+
ProcessCameraProvider.getInstance(baseContext).let {
131+
it.addListener(
132+
{
133+
cameraProvider = it.get()
134+
135+
bindCameraUseCases()
136+
},
137+
Dispatchers.Main.asExecutor()
138+
)
139+
}
140+
}
141+
}
142+
143+
private fun bindCameraUseCases() {
144+
// TODO: TO BE IMPLEMENTED
145+
}
146+
```
147+
148+
4. Implement the above `bindCameraUseCases()` method:
149+
150+
```kotlin
151+
private fun bindCameraUseCases() {
152+
val cameraProvider = cameraProvider
153+
?: throw IllegalStateException("Camera initialization failed.")
154+
155+
val cameraSelector =
156+
CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_FRONT).build()
157+
158+
// Only using the 4:3 ratio because this is the closest to MediaPipe models
159+
val resolutionSelector =
160+
ResolutionSelector.Builder()
161+
.setAspectRatioStrategy(AspectRatioStrategy.RATIO_4_3_FALLBACK_AUTO_STRATEGY)
162+
.build()
163+
val targetRotation = viewBinding.viewFinder.display.rotation
164+
165+
// Preview usecase.
166+
preview = Preview.Builder()
167+
.setResolutionSelector(resolutionSelector)
168+
.setTargetRotation(targetRotation)
169+
.build()
170+
171+
// Must unbind the use-cases before rebinding them
172+
cameraProvider.unbindAll()
173+
174+
try {
175+
// A variable number of use-cases can be passed here -
176+
// camera provides access to CameraControl & CameraInfo
177+
camera = cameraProvider.bindToLifecycle(
178+
this, cameraSelector, preview,
179+
)
180+
181+
// Attach the viewfinder's surface provider to preview use case
182+
preview?.surfaceProvider = viewBinding.viewFinder.surfaceProvider
183+
} catch (exc: Exception) {
184+
Log.e(TAG, "Use case binding failed", exc)
185+
}
186+
}
187+
```
188+
189+
5. Add a [companion object](https://kotlinlang.org/docs/object-declarations.html#companion-objects) to `MainActivity.kt` and declare a `TAG` constant value for `Log` calls to work correctly. This companion object comes handy for us to define all the constants and shared values accessible across the entire class.
190+
191+
```kotlin
192+
companion object {
193+
private const val TAG = "MainActivity"
194+
}
195+
```
196+
197+
In the next chapter, we will build and run the app to make sure the camera works well.

‎content/learning-paths/smartphones-and-mobile/build-android-selfie-app-using-mediapipe-multimodality/3-camera-permission.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
title: Handle camera permission
3+
weight: 3
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Run the app on your device
10+
11+
1. Connect your Android device to your computer via a USB **data** cable. If this is your first time running and debugging Android apps, follow [this guide](https://developer.android.com/studio/run/device#setting-up) and double check this checklist:
12+
13+
1. You have enabled **USB debugging** on your Android device following [this doc](https://developer.android.com/studio/debug/dev-options#Enable-debugging).
14+
15+
2. You have confirmed by tapping "OK" on your Android device when an **"Allow USB debugging"** dialog pops up, and checked "Always allow from this computer".
16+
17+
![Allow USB debugging dialog](https://ftc-docs.firstinspires.org/en/latest/_images/AllowUSBDebugging.jpg)
18+
19+
20+
2. Make sure your device model name and SDK version correctly show up on the top right toolbar. Click the **"Run"** button to build and run, as described [here](https://developer.android.com/studio/run).
21+
22+
3. After waiting for a while, you should be seeing success notification in Android Studio and the app showing up on your Android device.
23+
24+
4. However, the app shows only a black screen while printing error messages in your [Logcat](https://developer.android.com/tools/logcat) which looks like this:
25+
26+
```
27+
2024-11-20 11:15:00.398 18782-18818 Camera2CameraImpl com.example.holisticselfiedemo E Camera reopening attempted for 10000ms without success.
28+
2024-11-20 11:30:13.560 667-707 BufferQueueProducer pid-667 E [SurfaceView - com.example.holisticselfiedemo/com.example.holisticselfiedemo.MainActivity#0](id:29b00000283,api:4,p:2657,c:667) queueBuffer: BufferQueue has been abandoned
29+
2024-11-20 11:36:13.100 20487-20499 isticselfiedem com.example.holisticselfiedemo E Failed to read message from agent control socket! Retrying: Bad file descriptor
30+
2024-11-20 11:43:03.408 2709-3807 PackageManager pid-2709 E Permission android.permission.CAMERA isn't requested by package com.example.holisticselfiedemo
31+
```
32+
33+
5. Worry not. This is expected behavior because we haven't correctly configured this app's [permissions](https://developer.android.com/guide/topics/permissions/overview) yet, therefore Android OS restricts this app's access to camera features due to privacy reasons.
34+
35+
## Request camera permission at runtime
36+
37+
1. Navigate to `manifest.xml` in your `app` subproject's `src/main` path. Declare camera hardware and permission by inserting the following lines into the `<manifest>` element. Make sure it's **outside** and **above** `<application>` element.
38+
39+
```xml
40+
<uses-feature
41+
android:name="android.hardware.camera"
42+
android:required="true" />
43+
<uses-permission android:name="android.permission.CAMERA" />
44+
```
45+
46+
2. Navigate to `strings.xml` in your `app` subproject's `src/main/res/values` path. Insert the following lines of text resources, which will be used later.
47+
48+
```xml
49+
<string name="permission_request_camera_message">Camera permission is required to recognize face and hands</string>
50+
<string name="permission_request_camera_rationale">To grant Camera permission to this app, please go to system settings</string>
51+
```
52+
53+
3. Navigate to `MainActivity.kt` and add the following permission related values to companion object:
54+
55+
```kotlin
56+
// Permissions
57+
private val PERMISSIONS_REQUIRED = arrayOf(Manifest.permission.CAMERA)
58+
private const val REQUEST_CODE_CAMERA_PERMISSION = 233
59+
```
60+
61+
4. Add a new method named `hasPermissions()` to check on runtime whether camera permission has been granted:
62+
63+
```kotlin
64+
private fun hasPermissions(context: Context) = PERMISSIONS_REQUIRED.all {
65+
ContextCompat.checkSelfPermission(context, it) == PackageManager.PERMISSION_GRANTED
66+
}
67+
```
68+
69+
5. Add a condition check in `onCreate()` wrapping `setupCamera()` method, to request camera permission on runtime.
70+
71+
```kotlin
72+
if (!hasPermissions(baseContext)) {
73+
requestPermissions(
74+
arrayOf(Manifest.permission.CAMERA),
75+
REQUEST_CODE_CAMERA_PERMISSION
76+
)
77+
} else {
78+
setupCamera()
79+
}
80+
```
81+
82+
6. Override `onRequestPermissionsResult` method to handle permission request results:
83+
84+
```kotlin
85+
override fun onRequestPermissionsResult(
86+
requestCode: Int,
87+
permissions: Array<out String>,
88+
grantResults: IntArray
89+
) {
90+
when (requestCode) {
91+
REQUEST_CODE_CAMERA_PERMISSION -> {
92+
if (PackageManager.PERMISSION_GRANTED == grantResults.getOrNull(0)) {
93+
setupCamera()
94+
} else {
95+
val messageResId =
96+
if (shouldShowRequestPermissionRationale(Manifest.permission.CAMERA))
97+
R.string.permission_request_camera_rationale
98+
else
99+
R.string.permission_request_camera_message
100+
Toast.makeText(baseContext, getString(messageResId), Toast.LENGTH_LONG).show()
101+
}
102+
}
103+
else -> super.onRequestPermissionsResult(requestCode, permissions, grantResults)
104+
}
105+
}
106+
```
107+
108+
## Verify camera permission
109+
110+
1. Rebuild and run the app. Now you should be seeing a dialog pops up requesting camera permissions!
111+
112+
2. Tap `Allow` or `While using the app` (depending on your Android OS versions), then you should be seeing your own face in the camera preview. Good job!
113+
114+
{{% notice Tip %}}
115+
Sometimes you might need to restart the app to observe the permission change take effect.
116+
{{% /notice %}}
117+
118+
In the next chapter, we will introduce MediaPipe vision solutions.

0 commit comments

Comments
 (0)