You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Run LLMs locally on Raspberry Pi 5 for Edge AI
3
+
3
4
weight: 2
4
5
5
6
### FIXED, DO NOT MODIFY
@@ -8,66 +9,65 @@ layout: learningpathall
8
9
9
10
## Overview
10
11
11
-
This Learning Path walks you through deploying an efficient large language model (LLM) locally on the Raspberry Pi 5, powered by an Arm Cortex-A76 CPU. This will allow you to control your smart home using natural language, without relying on cloud services. With rapid advances in Generative AI and the power of Arm Cortex-A processors, you can now run advanced language models directly in your home on the Raspberry Pi 5.
12
+
This Learning Path walks you through deploying an efficient large language model (LLM) locally on the Raspberry Pi 5, powered by an Arm Cortex-A76 CPU. This setup enables you to control your smart home using natural language without relying on cloud services. With rapid advances in generative AI and the power of Arm Cortex-A processors, you can now run advanced language models directly in your home on the Raspberry Pi 5.
12
13
13
-
You will create a fully local, privacy-first smart home system that leverages the strengths of Arm Cortex-A architecture. The system can achieve 15+ tokens per second inference speeds using optimized models like TinyLlama and Qwen, while maintaining the energy efficiency that makes Arm processors a good fit for always-on applications.
14
+
You will create a fully local, privacy-first smart home system that leverages the strengths of Arm Cortex-A architecture. The system can achieve 15+ tokens per second inference speeds using optimized models like TinyLlama and Qwen, while maintaining the energy efficiency that makes Arm processors well suited for always-on applications.
14
15
15
-
## Why Arm Cortex-A for Edge AI?
16
+
## Why Arm Cortex-A76 makes Raspberry Pi 5 ideal for Edge AI
16
17
17
18
The Raspberry Pi 5's Arm Cortex-A76 processor can manage high-performance computing tasks like AI inference. Key architectural features include:
18
19
19
-
-The **superscalar architecture** allows the processor to execute multiple instructions in parallel, improving throughput for compute-heavy tasks.
20
-
-**128-bit NEON SIMD support** accelerates matrix and vector operations, which are common in the inner loops of language model inference.
21
-
-The **multi-level cache hierarchy** helps reduce memory latency and improves data access efficiency during runtime.
22
-
-The **thermal efficiency** enables sustained performance without active cooling, making it ideal for compact or always-on smart home setups.
20
+
-**Superscalar architecture**: Executes multiple instructions in parallel, improving throughput for compute-heavy tasks
21
+
-**128-bit NEON SIMD support**: Accelerates matrix and vector operations, common in the inner loops of language model inference
22
+
-**Multi-level cache hierarchy**: Reduces memory latency and improves data access efficiency during runtime
23
+
-**Thermal efficiency**: Enables sustained performance without active cooling, making it ideal for compact or always-on smart home setups
23
24
24
-
These characteristics make the Raspberry Pi 5 well-suited for workloads like smart home assistants, where responsiveness, efficiency, and local processing are important. Running LLMs locally on Arm-based devices brings several practical benefits. Privacy is preserved, since conversations and routines never leave the device. With optimized inference, the system can offer responsiveness under 100 ms, even on resource-constrained hardware. It remains fully functional in offline scenarios, continuing to operate when internet access is unavailable. Developers also gain flexibility to customize models and automations. Additionally, software updates and an active ecosystem continue to improve performance over time.
25
+
These characteristics make the Raspberry Pi 5 wellsuited for workloads like smart home assistants, where responsiveness, efficiency, and local processing are important. Running LLMs locally on Arm-based devices brings several practical benefits. Privacy is preserved, since conversations and routines never leave the device. With optimized inference, the system can offer responsiveness under 100 ms, even on resource-constrained hardware. It remains fully functional in offline scenarios, continuing to operate when internet access is unavailable. Developers also gain flexibility to customize models and automations. Additionally, software updates and an active ecosystem continue to improve performance over time.
25
26
26
-
## Arm Ecosystem Advantages
27
+
## Leverage the Arm ecosystem for Raspberry Pi Edge AI
27
28
28
29
For the stack in this setup, Raspberry Pi 5 benefits from the extensive developer ecosystem:
29
30
30
31
- Optimized compilers including GCC and Clang with Arm-specific enhancements
31
32
- Native libraries such as gpiozero and lgpio are optimized for Raspberry Pi
32
-
- Community support from open-source projects where developers are contributing Arm-optimized code
33
-
-Arm maintains a strong focus on backward compatibility, which reduces friction when updating kernels or deploying across multiple Arm platforms
33
+
- Community support from open-source projects where developers contribute Arm-optimized code
34
+
-Backward compatibility in Arm architecture reduces friction when updating kernels or deploying across platforms
34
35
- The same architecture powers smartphones, embedded controllers, edge devices, and cloud infrastructure—enabling consistent development practices across domains
35
36
36
-
## Performance Benchmarks on Raspberry Pi 5
37
+
## Performance benchmarks on Raspberry Pi 5
37
38
38
39
The table below shows inference performance for several quantized models running on a Raspberry Pi 5. Measurements reflect single-threaded CPU inference with typical prompt lengths and temperature settings suitable for command-based interaction.
What does this table tell us? Here are some performance insights:
50
-
51
-
- Qwen 0.5B and TinyLlama 1.1B deliver fast token generation and low average latency, making them suitable for real-time interactions like voice-controlled smart home commands.
52
-
- DeepSeek-Coder 1.3B and Gemma 2B trade off some speed for improved language understanding, which can be useful for more complex task execution or context-aware prompts.
53
-
- DeepSeek-R1 7B offers advanced reasoning capabilities with acceptable latency, which may be viable for offline summarization, planning, or low-frequency tasks.
51
+
- Qwen 0.5B and TinyLlama 1.1B deliver fast token generation and low average latency, making them suitable for real-time interactions such as voice-controlled smart home commands
52
+
- DeepSeek-Coder 1.3B and Gemma 2B trade some speed for improved language understanding, which can be useful for complex tasks or context-aware prompts
53
+
- DeepSeek-R1 7B offers advanced reasoning capabilities with acceptable latency, which may be viable for offline summarization, planning, or low-frequency tasks
54
54
55
-
## Supported Arm-Powered Devices
55
+
## Supported Arm-powered devices
56
56
57
-
This Learning Path focuses on the Raspberry Pi 5, but you can adapt the concepts and code to other Arm-powered devices:
57
+
This Learning Path focuses on the Raspberry Pi 5, but you can adapt the concepts and code to other Arm-powered devices.
58
58
59
-
###Recommended Platforms
59
+
## Recommended platforms
60
60
61
-
| Platform | CPU | RAM | GPIO Support| Model Size Suitability|
|**Raspberry Pi 5**| Arm Cortex-A76 quad-core @ 2.4GHz | Up to 16GB | Native `lgpio` (high-performance) | Large models (8–16GB) |
64
+
|**Raspberry Pi 4**| Arm Cortex-A72 quad-core @ 1.8GHz | Up to 8GB | Compatible with `gpiozero`| Small to mid-size models |
65
+
|**Other Arm devices**| Arm Cortex-A | 4GB min (8GB+ recommended) | Requires physical GPIO pins | Varies by RAM |
66
66
67
-
Additionally, the platform must:
67
+
Additionally, the platform must meet the following requirements:
68
68
69
69
- GPIO pins available for hardware control
70
-
-Use Python 3.8 or newer
70
+
- Python 3.8 or newer
71
71
- Ability to run [Ollama](https://ollama.com/)
72
72
73
-
Continue to the next section to start building a smart home system that highlights how Arm-based processors can enable efficient, responsive, and private AI applications at the edge.
73
+
In the next section, you’ll set up the software dependencies needed to start building your privacy-first smart home system on Raspberry Pi 5.
title: Set up software dependencies on Raspberry Pi 5 for Ollama and LLMs
3
3
weight: 3
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
8
9
+
## Overview
10
+
11
+
In this section, you’ll prepare your Raspberry Pi 5 by installing Python, required libraries, and Ollama, so you can run large language models (LLMs) locally.
12
+
9
13
{{% notice Note %}}
10
-
This guide assumes you have set up your Raspberry Pi with Raspberry Pi OS and network connectivity. For Raspberry Pi 5 setup help, see:[Raspberry Pi Getting Started](https://www.raspberrypi.com/documentation/)
14
+
This Learning Path assumes you have set up your Raspberry Pi with Raspberry Pi OS and network connectivity. For Raspberry Pi 5 setup support, see [Raspberry Pi Getting Started](https://www.raspberrypi.com/documentation/).
11
15
{{% /notice %}}
12
16
13
-
## Connect to Your Raspberry Pi 5
17
+
## Connect to your Raspberry Pi 5
14
18
15
-
### Option 1: Using a display
19
+
### Option 1: Use a display
16
20
17
-
The easiest way to work on your Raspberry Pi is connecting it to an external display through one of the microHDMI ports. This setup also requires a keyboard and mouse to navigate.
21
+
The easiest way to work on your Raspberry Pi is by connecting it to an external display through one of the micro‑HDMI ports. This setup also requires a keyboard and mouse.
18
22
19
-
### Option 2: Using SSH
23
+
### Option 2: Use SSH
20
24
21
-
You can also use SSH to access the terminal. To use this approach you need to know the IP address of your device. Ensure your Raspberry Pi 5 connects to the same network as your host computer. Access your device remotely via SSH using the terminal or any SSH client.
25
+
You can also use SSH to access the terminal. To use this approach, you need to know the IP address of your device. Ensure your Raspberry Pi 5 is on the same network as your host computer. Access your device remotely via SSH using the terminal or any SSH client.
22
26
23
27
Replace `<user>` with your Pi's username (typically `pi`), and `<pi-ip>` with your Raspberry Pi 5's IP address.
24
28
25
29
```bash
26
30
ssh <user>@<pi-ip>
27
31
```
28
32
29
-
## Set up the dependencies
33
+
## Install Python and system dependencies
30
34
31
35
Create a directory called `smart-home` in your home directory and navigate into it:
32
36
33
37
```bash
34
-
mkdir $HOME/smart-home
35
-
cd$HOME/smart-home
38
+
mkdir -p "$HOME/smart-home"
39
+
cd"$HOME/smart-home"
36
40
```
37
41
38
-
The Raspberry Pi 5 includes Python 3 pre-installed, but you need additional packages:
42
+
The Raspberry Pi 5 includes Python 3 preinstalled, but you need additional packages:
The next step is to create and activate a Python virtual environment. This approach keeps project dependencies isolated and prevents conflicts with system-wide packages:
51
+
Create and activate a Python virtual environment to isolate project dependencies:
Install Ollama using the official installation script for Linux:
63
67
@@ -70,27 +74,29 @@ Verify the installation:
70
74
```bash
71
75
ollama --version
72
76
```
73
-
If installation was successful, the output from the command should match that below.
77
+
78
+
If installation was successful, the output should be similar to:
79
+
74
80
```output
75
81
ollama version is 0.11.4
76
82
```
77
83
78
-
## Download and Test a Language Model
84
+
## Run a test LLM with Ollama on Raspberry Pi 5
79
85
80
-
Ollama supports various models. This guide uses deepseek-r1:7b as an example, but you can also use `tinyllama:1.1b`, `qwen:0.5b`, `gemma2:2b`, or `deepseek-coder:1.3b`.
86
+
Ollama supports various models. This guide uses `deepseek-r1:7b` as an example, but you can also use `tinyllama:1.1b`, `qwen:0.5b`, `gemma2:2b`, or `deepseek-coder:1.3b`.
81
87
82
-
The `run` command will set up the model automatically. You will see download progress in the terminal, followed by the interactive prompt when ready.
88
+
The `run` command sets up the model automatically. You will see download progress in the terminal, followed by an interactive prompt when ready.
83
89
84
90
```bash
85
91
ollama run deepseek-r1:7b
86
92
```
87
93
88
94
{{% notice Troubleshooting %}}
89
-
If you run into issues with the model download, here are some things to check:
95
+
If you run into issues with the model download, try the following:
90
96
91
-
- Confirm internet access and sufficient storage space on your microSD card
92
-
- Try downloading smaller models like `qwen:0.5b` or `tinyllama:1.1b` if you encounter memory issues. 16 GB of RAM is sufficient for running smaller to medium-sized language models. Very large models may require more memory or run slower.
93
-
- Clear storage or connect to a more stable network if errors occur
97
+
- Confirm internet access and sufficient storage space on your microSD card.
98
+
- Try smaller models like `qwen:0.5b` or `tinyllama:1.1b` if you encounter memory issues. 16 GB of RAM is sufficient for small to mediummodels; very large models may require more memory or run slower.
99
+
- Clear storage or connect to a more stable network if errors occur.
94
100
{{% /notice %}}
95
101
96
-
With the model set up through `ollama`, move on to the next section to start configuring the hardware.
102
+
With the model set up through Ollama, move on to the next section to start configuring the hardware.
title: Test Raspberry Pi 5 GPIO pins for smart home devices
3
3
weight: 4
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
8
9
-
The next step is to test the GPIO functionality. In this section, you will configure a LED light to simulate a smart-home device.
9
+
## Overview
10
10
11
-
## Verify GPIO Functionality
11
+
The next step is to test the GPIO functionality. In this section, you configure an LED light to simulate a smart home device.
12
12
13
-
Bring out your electronics components. Connect the anode (long leg) of an LED in series with a 220Ω resistor to GPIO 17 (physical pin 11). Connect the cathode (short leg) to a ground (GND) pin. See image below for the full setup:
13
+
## Verify GPIO setup on Raspberry Pi 5
14
14
15
-
![Raspberry Pi connected to a breadboard with a green LED and jumper wires](pin_layout.jpg"Raspberry Pi connected to a breadboard with a green LED and jumper wires")
15
+
Gather your electronic components. Connect the anode (long leg) of an LED in series with a 220Ω resistor to GPIO 17 (physical pin 11). Connect the cathode (short leg) to a ground (GND) pin.
16
+
17
+
See the image below for the full setup:
18
+
19
+
![Raspberry Pi connected to a breadboard with a green LED and jumper wires alt-text#center](pin_layout.jpg"Raspberry Pi connected to a breadboard with a green LED and jumper wires")
16
20
17
21
Create a Python script named `testgpio.py`:
18
22
@@ -21,7 +25,7 @@ cd $HOME/smart-home
21
25
vim testgpio.py
22
26
```
23
27
24
-
Copy this code into the file:
28
+
Add the following code to the file:
25
29
26
30
```python
27
31
#!/usr/bin/env python3
@@ -32,7 +36,7 @@ from gpiozero.pins.lgpio import LGPIOFactory
32
36
# Set lgpio backend for Raspberry Pi 5
33
37
Device.pin_factory = LGPIOFactory()
34
38
35
-
#Setup GPIO pin 17
39
+
#Set up GPIO pin 17
36
40
pin1 = LED(17)
37
41
38
42
try:
@@ -52,19 +56,20 @@ python testgpio.py
52
56
The LED should blink every two seconds. If you observe this behavior, your GPIO setup works correctly.
53
57
54
58
{{% notice Troubleshooting %}}
55
-
If you run into issues with the hardware setup, here are some things to check:
56
-
- Try fixing missing dependencies by running the following command:
57
-
```bash
58
-
sudo apt-get install -f
59
-
```
60
-
- If you're running into GPIO permission issues, run Python scripts with `sudo` or add your user to the `gpio` group. Don't forget to log out for the changes to take effect.
61
-
```bash
62
-
sudo usermod -a -G gpio $USER
63
-
```
59
+
If you run into issues with the hardware setup, check the following:
60
+
61
+
- Fix missing dependencies with:
62
+
```bash
63
+
sudo apt-get install -f
64
+
```
65
+
- If you encounter GPIO permission issues, run Python scripts with `sudo` or add your user to the `gpio` group. Don’t forget to log out for the changes to take effect:
66
+
```bash
67
+
sudo usermod -a -G gpio $USER
68
+
```
64
69
- Double-check wiring and pin numbers using the Raspberry Pi 5 pinout diagram
65
70
- Ensure proper LED and resistor connections
66
71
- Verify GPIO enablement in `raspi-config` if needed
67
72
- Use a high-quality power supply
68
73
{{% /notice %}}
69
74
70
-
With a way to control devices using GPIO pins, you can move on to the next section to interact with them using language models and the user interface.
75
+
With GPIO pins working, you can now move on to the next section to interact with devices using language models and the user interface.
0 commit comments