🚗 End-to-End Baseline Deployment Guide

Goal: Get the RC car moving with visual servoing (camera → policy → motors) using the FP32 baseline model on Raspberry Pi 5.

Philosophy: Establish complete working system first, then optimize.

📋 Deployment Phases

Phase 1: Camera Integration ✅ NEXT

Objective: Verify camera capture and inference pipeline
Hardware: Pi 5 + Camera (CSI or USB)
Expected: Real-time inference on live camera feed

Phase 2: Arduino Communication

Objective: Send control signals to Arduino
Hardware: Pi 5 + Arduino (USB serial)
Expected: Arduino receives steering/throttle commands

Phase 3: Full Control Loop

Objective: Complete visual servoing system
Hardware: Full system (Pi + Camera + Arduino + RC car)
Expected: WHEELS MOVE based on camera input! 🎉

Phase 4: Testing & Baseline Metrics

Objective: Measure real-world performance
Hardware: Full system on track
Expected: Documented baseline performance for future comparison

🎥 Phase 1: Camera Integration

Step 1.1: Identify Your Camera

Run this on the Pi:

# Check for USB cameras
ls -l /dev/video*

# Check for CSI cameras
vcgencmd get_camera

# Test with libcamera (Pi Camera Module)
libcamera-hello --list-cameras

Expected Output:

USB camera: /dev/video0, /dev/video1, etc.
CSI camera: "Detected camera 0" from libcamera

Step 1.2: Test Camera Capture

For USB Camera:

cd ~/EDTH2025/Erewhon/src/robots/rover
source ~/EDTH2025/Erewhon/venv/bin/activate
export PYTHONPATH=~/EDTH2025/Erewhon/src/policies/ACT/lerobot/src:$PYTHONPATH

python src/inference/act_inference_quantized.py \
    --checkpoint models/best_model.pth \
    --camera_id 0 \
    --test_duration 10

For CSI Camera (Pi Camera Module):

# Use picamera2 library
python src/inference/act_inference_quantized.py \
    --checkpoint models/best_model.pth \
    --camera_type picamera2 \
    --test_duration 10

What to Look For:

✅ Camera opens successfully
✅ Inference runs at ~30-40 FPS
✅ No dropped frames
✅ Steering/throttle predictions displayed

Step 1.3: Verify Image Quality

Check:

Resolution: 640x360 (or close)
Brightness: Not too dark/bright
Focus: Sharp, not blurry
FPS: 30+ frames per second

Adjust if needed:

# In the inference script, camera parameters:
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 360)
cap.set(cv2.CAP_PROP_FPS, 30)
cap.set(cv2.CAP_PROP_AUTO_EXPOSURE, 1)  # Manual exposure
cap.set(cv2.CAP_PROP_EXPOSURE, 100)     # Adjust as needed

🔌 Phase 2: Arduino Communication

Step 2.1: Connect Arduino to Pi

Physical Connection:

USB cable: Arduino → Raspberry Pi USB port
Power Arduino (if needed)

Verify Connection:

# Check serial ports
ls -l /dev/ttyUSB* /dev/ttyACM*

# Should see something like:
# /dev/ttyACM0  or  /dev/ttyUSB0

Step 2.2: Test Serial Communication

Run this on the Pi:

cd ~/EDTH2025/Erewhon/src/robots/rover
source ~/EDTH2025/Erewhon/venv/bin/activate

# Test Arduino communication (no camera yet)
python src/inference/act_inference_quantized.py \
    --checkpoint models/best_model.pth \
    --arduino_port /dev/ttyACM0 \
    --test_arduino

Expected:

✅ Serial port opens
✅ Arduino responds
✅ Test commands sent (steering: 0.0, throttle: 0.0)

Step 2.3: Verify Arduino Code

Your Arduino should:

Read serial commands: STEER:<value>,THROTTLE:<value>\n
Parse values: -1.0 to 1.0 range
Convert to PWM: 1000-2000 microseconds
Send to ESC/servo

Example Arduino code structure:

void loop() {
    if (Serial.available()) {
        String cmd = Serial.readStringUntil('\n');
        parseCommand(cmd);  // Extract steering, throttle
        updateMotors();     // Send PWM to ESC/servo
    }
}

🎮 Phase 3: Full Control Loop

Step 3.1: Camera + Arduino Test

Run the complete pipeline:

cd ~/EDTH2025/Erewhon/src/robots/rover
source ~/EDTH2025/Erewhon/venv/bin/activate
export PYTHONPATH=~/EDTH2025/Erewhon/src/policies/ACT/lerobot/src:$PYTHONPATH

python src/inference/act_inference_quantized.py \
    --checkpoint models/best_model.pth \
    --camera_id 0 \
    --arduino_port /dev/ttyACM0 \
    --control_freq 30

What Happens:

Camera captures frame (640x360)
Inference predicts steering + throttle
Commands sent to Arduino via serial
Arduino controls motors
WHEELS MOVE! 🚗💨

Step 3.2: Safety Checks

Before running on the car:

Lift the car: Wheels off the ground initially
Test steering range: Verify servo moves correctly
Test throttle range: Start with low values (0.0 to 0.3)
Emergency stop: Have keyboard interrupt ready (Ctrl+C)
Wireless: Use SSH + screen for remote control

Safety Mode:

# Start with limited throttle
python src/inference/act_inference_quantized.py \
    --checkpoint models/best_model.pth \
    --camera_id 0 \
    --arduino_port /dev/ttyACM0 \
    --control_freq 30 \
    --max_throttle 0.3  # Limit speed

Step 3.3: Track Testing

Setup:

Place car on the track (or similar environment to training)
Start the control loop
Observe behavior
Log performance

Monitor:

Control frequency (should be ~30 Hz)
Inference latency (should be ~25ms)
Steering smoothness
Throttle response
Track following accuracy

📊 Phase 4: Baseline Performance Metrics

Step 4.1: Log Data During Test

Metrics to Collect:

# Run with logging
python src/inference/act_inference_quantized.py \
    --checkpoint models/best_model.pth \
    --camera_id 0 \
    --arduino_port /dev/ttyACM0 \
    --control_freq 30 \
    --log_performance \
    --output_log results/baseline_run_$(date +%Y%m%d_%H%M%S).json

Data to Log:

Inference latency per frame
Camera FPS
Control loop frequency
Steering/throttle predictions
Dropped frames (if any)
Total runtime
Track completion (if applicable)

Step 4.2: Analyze Results

Key Metrics:

# After the run, analyze the log:
- Mean inference latency: [X] ms
- Control loop rate: [Y] Hz
- Camera capture rate: [Z] FPS
- Dropped frames: [N] / [Total]
- Track completion: [Success/Fail]
- Distance traveled: [meters]
- Average speed: [m/s]

Step 4.3: Document Baseline

Create: results/baseline_full_system_YYYYMMDD.md

Include:

Hardware setup (Pi 5 + Camera + Arduino + RC car specs)
Software versions
Model details (FP32, 960MB)
Performance metrics (from logs)
Video recording (if possible)
Issues encountered
Lessons learned

⚠️ Troubleshooting

Camera Issues

Problem: Camera not detected

# Check permissions
sudo usermod -a -G video $USER
# Logout and login again

# Test with simple capture
python -c "import cv2; cap = cv2.VideoCapture(0); print('Camera:', cap.read()[0])"

Problem: Low FPS

Reduce resolution: 320x180 or 480x270
Check camera power
Check USB bandwidth (use USB 3.0 if available)

Arduino Issues

Problem: Port not found

# Check permissions
sudo usermod -a -G dialout $USER
# Logout and login again

# Check port
ls -l /dev/ttyACM* /dev/ttyUSB*

Problem: Commands not received

Check baud rate (115200 default)
Verify Arduino code is running
Test with serial monitor first

Control Loop Issues

Problem: Latency too high

Reduce camera resolution
Lower control frequency (20 Hz instead of 30 Hz)
Check for thermal throttling: vcgencmd measure_temp

Problem: Erratic behavior

Verify training data matches test environment
Check action normalization/denormalization
Reduce throttle limit initially

🎯 Success Criteria

Minimum Viable System ✅

Baseline Performance 📊

End-to-end latency < 50ms
Control loop runs at 20-30 Hz
Car follows track (even if imperfectly)
Data logged for analysis
Documented in results/

Ready for Optimization 🚀

Once baseline is established:

✅ Know exactly what to optimize
✅ Have performance metrics to compare against
✅ Can measure improvement from Hailo deployment
✅ Can compare different approaches objectively

🔄 Next Steps After Baseline

Analyze performance bottlenecks
- Camera capture time?
- Inference time?
- Serial communication time?
- Arduino processing time?
ONNX Export
- Convert model to hardware-agnostic format
- Validate accuracy matches PyTorch
Hailo Quantization
- INT8 quantization with calibration data
- Compile for Hailo8L NPU
Hailo Deployment
- 10-20x speedup expected
- Compare against baseline
- Document improvement
Publish Results
- Compare FP32 baseline vs Hailo INT8
- Write paper/blog post
- Share learnings with community

Remember: The goal is to see the wheels move based on camera input! Everything else is optimization. 🎉

Current Status: Ready for Phase 1 (Camera Integration)
Next Command: Test camera on Pi → See Phase 1, Step 1.1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚗 End-to-End Baseline Deployment Guide

📋 Deployment Phases

Phase 1: Camera Integration ✅ NEXT

Phase 2: Arduino Communication

Phase 3: Full Control Loop

Phase 4: Testing & Baseline Metrics

🎥 Phase 1: Camera Integration

Step 1.1: Identify Your Camera

Step 1.2: Test Camera Capture

Step 1.3: Verify Image Quality

🔌 Phase 2: Arduino Communication

Step 2.1: Connect Arduino to Pi

Step 2.2: Test Serial Communication

Step 2.3: Verify Arduino Code

🎮 Phase 3: Full Control Loop

Step 3.1: Camera + Arduino Test

Step 3.2: Safety Checks

Step 3.3: Track Testing

📊 Phase 4: Baseline Performance Metrics

Step 4.1: Log Data During Test

Step 4.2: Analyze Results

Step 4.3: Document Baseline

⚠️ Troubleshooting

Camera Issues

Arduino Issues

Control Loop Issues

🎯 Success Criteria

Minimum Viable System ✅

Baseline Performance 📊

Ready for Optimization 🚀

🔄 Next Steps After Baseline

FilesExpand file tree

END_TO_END_DEPLOYMENT.md

Latest commit

History

END_TO_END_DEPLOYMENT.md

File metadata and controls

🚗 End-to-End Baseline Deployment Guide

📋 Deployment Phases

Phase 1: Camera Integration ✅ NEXT

Phase 2: Arduino Communication

Phase 3: Full Control Loop

Phase 4: Testing & Baseline Metrics

🎥 Phase 1: Camera Integration

Step 1.1: Identify Your Camera

Step 1.2: Test Camera Capture

Step 1.3: Verify Image Quality

🔌 Phase 2: Arduino Communication

Step 2.1: Connect Arduino to Pi

Step 2.2: Test Serial Communication

Step 2.3: Verify Arduino Code

🎮 Phase 3: Full Control Loop

Step 3.1: Camera + Arduino Test

Step 3.2: Safety Checks

Step 3.3: Track Testing

📊 Phase 4: Baseline Performance Metrics

Step 4.1: Log Data During Test

Step 4.2: Analyze Results

Step 4.3: Document Baseline

⚠️ Troubleshooting

Camera Issues

Arduino Issues

Control Loop Issues

🎯 Success Criteria

Minimum Viable System ✅

Baseline Performance 📊

Ready for Optimization 🚀

🔄 Next Steps After Baseline