Skip to content

Commit fb603d6

Browse files
psiddhGithub Executorch
psiddh
and
Github Executorch
authored
Add README & DESIGN details (#123)
Co-authored-by: Github Executorch <[email protected]>
1 parent cf01cb2 commit fb603d6

File tree

2 files changed

+250
-0
lines changed

2 files changed

+250
-0
lines changed

‎Yolo/android/DESIGN-README,md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
2+
# BirdDetection App: High-Level Flow (YOLO + Android + Species Recognition)
3+
4+
[1] Android Camera Frame
5+
|
6+
v
7+
[2] Convert Camera Image (YUV/NV21) --> RGB Bitmap
8+
|
9+
v
10+
[3] Preprocess Bitmap for YOLO (resize, normalize)
11+
|
12+
v
13+
[4] YOLO Model Inference (ExecuTorch)
14+
|
15+
v
16+
[5] YOLO Output: List of Detections
17+
| - Each detection: [center_x, center_y, width, height, class scores...]
18+
| - For birds: class 14 (COCO) has highest score
19+
|
20+
v
21+
[6] Parse YOLO Output
22+
| - For each detection:
23+
| - If class == bird and confidence > threshold
24+
| - Convert [center_x, center_y, width, height] to [left, top, right, bottom]
25+
| - Scale coordinates to match actual image size
26+
|
27+
v
28+
[7] Non-Maximum Suppression (NMS)
29+
| - Remove overlapping boxes (keep highest confidence)
30+
|
31+
v
32+
[8] For Each Bird Bounding Box:
33+
| - Crop region from original Bitmap
34+
| - Preprocess crop for classifier (resize, normalize)
35+
| - Run Bird Species Classifier (ExecuTorch)
36+
| - Get species name and classifier confidence
37+
|
38+
v
39+
[9] Display Results in Android UI
40+
| - Draw bounding boxes and species labels on overlay
41+
| - Log sightings (thumbnail, species, confidence, timestamp)
42+
|
43+
v
44+
[10] User Interactions
45+
| - Start/End session
46+
| - View log of detected birds
47+
48+
## Summary of Each Step
49+
50+
- Camera Frame: Android captures a live image from the camera.
51+
- Image Conversion: Converts raw camera format (YUV/NV21) to RGB Bitmap for ML processing.
52+
- Preprocessing: Resizes and normalizes the bitmap to match YOLO model input requirements.
53+
- YOLO Inference: Runs the YOLO model using ExecuTorch to get object detections.
54+
- YOLO Output: Produces a list of bounding boxes and class scores for each detected object.
55+
- Parsing: Filters detections for birds, converts coordinates, and scales to image size.
56+
- NMS: Removes duplicate/overlapping detections.
57+
- Species Recognition: Crops each bird region, runs a classifier to identify species.
58+
- UI Display: Shows bounding boxes and species names, logs each sighting.
59+
- User Actions: Allows session management and viewing of bird logs.
60+
61+
62+
### Design flow gist
63+
64+
- YOLO finds birds in the camera image.
65+
- Android handles image conversion, UI, and session management.
66+
- The classifier identifies the bird species for each detected bird.

‎Yolo/android/README.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,185 @@
1+
# Overview
12

3+
## 🐦 Android Bird Detection App with ExecuTorch
4+
5+
A real-time bird detection and species identification Android app using YOLO + EfficientNet models deployed via Meta's ExecuTorch framework. The app provides session-based bird watching with automatic logging, thumbnails, and timestamps for backyard bird enthusiasts.
6+
Bird Detection Feature: Uses a two-stage pipeline where YOLO (COCO class 14) detects birds in camera frames, then EfficientNet classifier identifies the specific species from 525 possible bird types with 96.8% accuracy.
7+
8+
## Model Download and Conversion
9+
10+
### Step 1: Download Models
11+
12+
#### Bird Classifier Model
13+
14+
Install dependencies
15+
16+
```
17+
pip install transformers torch pillow executorch
18+
Download EfficientNet bird classifier
19+
python -c "
20+
from transformers import AutoImageProcessor, AutoModelForImageClassification
21+
import torch
22+
model_name = 'dennisjooo/Birds-Classifier-EfficientNetB2'
23+
processor = AutoImageProcessor.from_pretrained(model_name)
24+
model = AutoModelForImageClassification.from_pretrained(model_name)
25+
model.save_pretrained('./bird_classifier_model')
26+
processor.save_pretrained('./bird_classifier_model')
27+
print('Bird classifier downloaded')
28+
"
29+
```
30+
31+
### YOLO Detection Model
32+
33+
Install YOLOv8
34+
35+
```
36+
pip install ultralytics
37+
Download YOLO model
38+
python -c "
39+
from ultralytics import YOLO
40+
model = YOLO('yolov8n.pt') # nano version for mobile
41+
print('YOLO model downloaded')
42+
"
43+
```
44+
45+
## Model Conversion to ExecuTorch
46+
47+
### Step 2: Convert Models to .pte Format
48+
49+
#### Convert Bird Classifier
50+
51+
convert_bird_classifier.py
52+
53+
```
54+
import torch
55+
from transformers import AutoModelForImageClassification
56+
from torch.export import export
57+
from executorch.exir import to_edge_transform_and_lower
58+
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
59+
Load model
60+
model = AutoModelForImageClassification.from_pretrained('./bird_classifier_model')
61+
model.eval()
62+
Export to ExecuTorch
63+
example_input = torch.randn(1, 3, 224, 224)
64+
exported_program = export(model, (example_input,))
65+
edge_program = to_edge_transform_and_lower(
66+
exported_program,
67+
partitioner=[XnnpackPartitioner()]
68+
)
69+
et_program = edge_program.to_executorch()
70+
Save as .pte file
71+
with open("bird_classifier.pte", "wb") as f:
72+
et_program.write_to_file(f)
73+
print("Bird classifier converted to bird_classifier.pte")
74+
```
75+
76+
### Convert YOLO Model
77+
78+
convert_yolo.py
79+
80+
```
81+
from ultralytics import YOLO
82+
import torch
83+
from torch.export import export
84+
from executorch.exir import to_edge_transform_and_lower
85+
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
86+
Load YOLO model
87+
yolo = YOLO('yolov8n.pt')
88+
pytorch_model = yolo.model
89+
pytorch_model.eval()
90+
Export to ExecuTorch
91+
example_input = torch.randn(1, 3, 640, 640)
92+
exported_program = export(pytorch_model, (example_input,))
93+
edge_program = to_edge_transform_and_lower(
94+
exported_program,
95+
partitioner=[XnnpackPartitioner()]
96+
)
97+
et_program = edge_program.to_executorch()
98+
Save as .pte file
99+
with open("yolo_detector.pte", "wb") as f:
100+
et_program.write_to_file(f)
101+
print("YOLO model converted to yolo_detector.pte")
102+
```
103+
104+
### Generate Bird Species Names
105+
106+
extract_species_names.py
107+
108+
```
109+
from transformers import AutoModelForImageClassification
110+
import json
111+
model = AutoModelForImageClassification.from_pretrained('./bird_classifier_model')
112+
species_names = [model.config.id2label[i] for i in range(len(model.config.id2label))]
113+
with open('bird_species.json', 'w') as f:
114+
json.dump(species_names, f, indent=2)
115+
print(f"Saved {len(species_names)} bird species names to bird_species.json")
116+
```
117+
118+
## Deploying Models to Android
119+
120+
### Step 3: Deploy Models to Android Device
121+
122+
#### Copy Models to Android Assets
123+
124+
```
125+
Copy .pte files to Android assets directory
126+
cp bird_classifier.pte /path/to/android/app/src/main/assets/
127+
cp yolo_detector.pte /path/to/android/app/src/main/assets/
128+
cp bird_species.json /path/to/android/app/src/main/assets/
129+
```
130+
131+
### Alternative: Push via ADB (for testing)
132+
133+
Connect Android device and enable USB debugging
134+
135+
```
136+
adb devices
137+
Create directory on device
138+
adb shell mkdir -p /data/local/tmp/bird_detection/
139+
Push model files
140+
adb push bird_classifier.pte /data/local/tmp/bird_detection/
141+
adb push yolo_detector.pte /data/local/tmp/bird_detection/
142+
adb push bird_species.json /data/local/tmp/bird_detection/
143+
Verify files are transferred
144+
adb shell ls -la /data/local/tmp/bird_detection/
145+
```
146+
147+
### File Structure
148+
149+
```
150+
app/src/main/assets/
151+
├── bird_classifier.pte # EfficientNet bird species classifier (8.5MB)
152+
├── yolo_detector.pte # YOLOv8n bird detection model (6MB)
153+
└── bird_species.json # List of 525 bird species names
154+
```
155+
156+
## App Features
157+
158+
### Main Detection Screen
159+
160+
- Camera Preview: Real-time video feed from device camera
161+
- Live Detection: Green bounding boxes around detected birds with species labels
162+
- Session Controls: Start/Stop buttons for bird watching sessions
163+
164+
### Session Management
165+
166+
#### "Start Session" Button:
167+
168+
- Activates bird detection and logging
169+
- Changes camera from passive viewing to active detection mode
170+
- Begins collecting bird sightings with timestamps and thumbnails
171+
- Button turns red and displays "Stop Session"
172+
- "Stop Session" Button:
173+
- Deactivates detection and shows session summary
174+
- Displays total birds detected and unique species count
175+
- Button turns green and displays "Start Session"
176+
- Preserves collected data for viewing
177+
178+
### Bird Log Viewer
179+
180+
#### "View Logs" Button:
181+
182+
- Opens detailed session log showing all detected birds
183+
- Displays bird thumbnails, species names, detection times, and confidence scores
184+
- Organized as scrollable list with visual bird identification records
185+
- Useful for reviewing and verifying bird watching session results

0 commit comments

Comments
 (0)