Skip to content

Commit 45d9a13

Browse files
matt-cossinsmatt-cossins
authored
Edge AI (IoT/Client) Projects (#30)
added 4 EdgeAI projects
1 parent cb0f877 commit 45d9a13

File tree

4 files changed

+363
-0
lines changed

4 files changed

+363
-0
lines changed

‎Projects/Projects/Edge-AI-with-Ethos-U85-NPU.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
---
2+
title: "Edge AI with NPU: always-on-AI with ExecuTorch on Cortex-M55 + Ethos-U85 → Cortex-A"
3+
description: "The vision of Edge AI compute is to embed low-power intelligent sensing, perception, and decision systems everywhere. A low-power always-on-AI island continuously monitors sensory inputs to detect triggers. When a trigger is detected, it wakes up a more capable processor to carry out high-value inference, interaction, or control tasks."
4+
subjects:
5+
- "ML"
6+
- "Performance and Architecture"
7+
- "Embedded Linux"
8+
- "RTOS Fundamentals"
9+
requires-team:
10+
- "No"
11+
platform:
12+
- "IoT"
13+
- "Embedded and Microcontrollers"
14+
- "AI"
15+
sw-hw:
16+
- "Software"
17+
- "Hardware"
18+
support-level:
19+
- "Self-Service"
20+
- "Arm Ambassador Support"
21+
publication-date: 2025-11-27
22+
license:
23+
status:
24+
- "Published"
25+
donation:
26+
---
27+
28+
![educate_on_arm](../../images/Educate_on_Arm_banner.png)
29+
30+
31+
## Description
32+
33+
**Why is this important?**
34+
35+
The vision of Edge AI compute is to embed intelligent low-power sensing, perception, and decision systems everywhere (in homes, wearables, infrastructure) so devices can react to subtle cues, adapt to context, and wake higher-power systems only when needed. Rather than sending everything to the cloud or running full-scale models continuously, this Edge AI system operates as a layered hierarchy:
36+
- A low-power always-on-AI model continuously monitors sensory inputs (audio, motion, video) to detect triggers or anomalies.
37+
- When a trigger is detected, it wakes up a more capable processor (e.g. Cortex-A running a rich OS such as Linux) to carry out further tasks. This could be high-value inference, interaction, or control tasks. It could also involve connecting to other IoT devices or to a Neoverse cloud instance.
38+
39+
This architecture is key to bridging the gap between battery-constrained devices and rich AI services, making systems smarter, more efficient, and responsive without draining resources.
40+
41+
**Project Summary**
42+
43+
Using equipment such as the Alif Ensemble development kit (e.g. E6/E8, which includes Cortex-A, Cortex-M55, and Ethos-U85 cores - or E4 (M55+U85) + Raspberry Pi for Cortex-A), and the ExecuTorch framework, build an Edge AI prototype that implements:
44+
45+
1. A “wake-up” path: deploy a TOSA-compliant optimized model on the Cortex-M55 + Ethos-U85 pair to continuously monitor sensory signals (audio, motion, video) for wake-word, anomalies, or triggers.
46+
2. A subsequent workload path: when a trigger is detected, activate a Cortex-A core to perform more complex tasks, e.g. use an LLM optimised for CPU inference, connect to and manage other IoT devices, or connect to a Neoverse cloud instance for heavier inference.
47+
3. Evaluation and documentation: measure accuracy, latency, power consumption, robustness, and compare trade-offs between modalities (audio, video, motion). Demonstrate an end-to-end use case of your choice (e.g. smart assistant, anomaly alert system, gesture control, environment monitoring).
48+
49+
*Note that the Cortex-A32 included on the Alif DevKits will not be suitable for LLM inference. If using the onboard core for the project, target cloud/IoT connectivity. For LLM inference, consider connecting a Raspberry Pi 5 or similar.*
50+
51+
Example: Use a microphone input to detect “Hey Arm”. After wake-up, launch an optimised LLM on Raspberry Pi Cortex-A to answer questions or control local devices.
52+
53+
You are free to mix and match sensors, modalities, and tasks — as long as the core architecture (wake-on M55/U85, main task on A) is preserved.
54+
55+
Many of these DevKits come with additional Ethos-U55 NPUs onboard - feel free to be creative and distribute different tasks across the different NPUs - what use-cases and applications can you achieve?
56+
57+
## What will you use?
58+
You should either be familiar with, or willing to learn about, the following:
59+
- Programming: Python, C++, Embedded C
60+
- ExecuTorch, plus knowledge of model quantization, pruning, conversion. Use of the Vela compiler and TOSA.
61+
- Edge/Embedded development: bare-metal or RTOS (e.g. Zephyr), and embedded Linux (e.g. Yocto) or Raspberry Pi OS
62+
63+
64+
## Resources from Arm and our partners
65+
- Arm Developer: [Edge AI](https://developer.arm.com/edge-ai)
66+
- Learning Path: [Navigating Machine Learning with Ethos-U processors](https://learn.arm.com/learning-paths/microcontrollers/nav-mlek/)
67+
- Repository: [AI on Arm course](https://github.com/arm-university/AI-on-Arm)
68+
- Example Board: [Alif Ensemble DevKit E8](https://www.keil.arm.com/boards/alif-semiconductor-devkit-e8-gen-1-2558a7b/features/)
69+
- PyTorch Blog: [ExecuTorch support for Ethos-U85](https://pytorch.org/blog/pt-executorch-ethos-u85/)
70+
71+
## Support Level
72+
73+
This project is designed to be self-serve but comes with opportunity of some community support from Arm Ambassadors, who are part of the Arm Developer program. If you are not already part of our program, [click here to join](https://www.arm.com/resources/developer-program?#register).
74+
75+
## Benefits
76+
77+
Standout project contributions will result in digital badges for CV building, recognised by Arm Talent Acquisition. We are currently discussing with national agencies the potential for funding streams for Arm Developer Labs projects, which would flow to you, not us.
78+
79+
80+
To receive the benefits, you must show us your project through our [online form](https://forms.office.com/e/VZnJQLeRhD). Please do not include any confidential information in your contribution. Additionally if you are affiliated with an academic institution, please ensure you have the right to share your material.

‎Projects/Projects/Ethos-U85-NPU-Applications.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
title: "Ethos-U85 NPU Applications with TOSA Model Explorer: Exploring Next-Gen Edge AI Inference"
3+
description: "Push the limits of Edge AI by deploying the heaviest inference applications possible on Ethos-U85. Students will explore transformer-based and TOSA-optimized workloads that demonstrate performance levels on the next-gen of Ethos NPUs."
4+
subjects:
5+
- "ML"
6+
- "Performance and Architecture"
7+
requires-team:
8+
- "No"
9+
platform:
10+
- "IoT"
11+
- "Embedded and Microcontrollers"
12+
- "AI"
13+
sw-hw:
14+
- "Software"
15+
- "Hardware"
16+
support-level:
17+
- "Self-Service"
18+
- "Arm Ambassador Support"
19+
publication-date: 2025-11-27
20+
license:
21+
status:
22+
- "Published"
23+
donation:
24+
---
25+
26+
![educate_on_arm](../../images/Educate_on_Arm_banner.png)
27+
28+
## Description
29+
30+
**Why is this important?**
31+
32+
The Arm Ethos-U85 NPU represents a major leap in bringing *heavy inference* to constrained embedded systems. With its full transformer operator support, expanded MAC throughput, and native TOSA compatibility, the Ethos-U85 enables developers to deploy models and workloads that were previously too intensive for MCU-class devices.
33+
34+
This project challenges you to explore the boundaries of what’s possible on Ethos-U85. The goal is to demonstrate inference performance and model complexity that is now achievable due to the architectural improvements and transformer acceleration capabilities of the Ethos-U85.
35+
36+
[Ethos-U85 Launch](https://newsroom.arm.com/blog/ethos-u85)
37+
38+
**Project Summary**
39+
40+
Using hardware such as the Alif Ensemble E4/E6/E8 DevKits (all include Ethos-U85) or a comparable platform or Arm Fixed Virtual Platform Corstone-320, your task is to design and benchmark an advanced edge inference application that exploits the Ethos-U85’s compute and transformer capabilities.
41+
42+
Your project should include:
43+
44+
1. Model Deployment and Optimization
45+
Select a computationally intensive model — ideally transformer-based or multi-branch convolutional — and deploy it on the Ethos-U85 using:
46+
- The TOSA Model Explorer extension to inspect and adapt unsupported or experimental models for TOSA compliance.
47+
- The Vela compiler for optimization.
48+
49+
These tools can be used to:
50+
- Convert and visualize model graphs in TOSA format.
51+
- Identify unsupported operators.
52+
- Modify or substitute layers for compatibility using the Flatbuffers schema before re-exporting.
53+
- Run Vela for optimized compilation targeting Ethos-U85.
54+
55+
2. Application Demonstration
56+
Implement a working example that highlights the Ethos-U85’s strengths in real-world inference. Possible categories include:
57+
- Transformers on Edge: lightweight BERT, ViT, or audio transformers (e.g. speech or sound event classification).
58+
- High-resolution Vision: semantic segmentation, object detection on large input sizes, or multi-head perception networks.
59+
- Multi-modal Fusion: combining audio, image, or sensor streams for contextual understanding.
60+
61+
3. Analysis and Benchmarking
62+
Report quantitative results on:
63+
- Inference latency, throughput (FPS or tokens/s), and memory footprint.
64+
- Power efficiency under load (optional).
65+
- Comparative performance versus Ethos-U55/U65 (use available benchmarks for reference or utilise the other Ethos-U NPUs provided in the Alif DevKits).
66+
- The effect of TOSA optimization — demonstrate measurable improvements from graph conversion and operator fusion.
67+
68+
---
69+
70+
## What kind of projects should you target?
71+
72+
To clearly demonstrate the leap from Ethos-U55/U65 to U85, choose projects that meet at least one of the following criteria:
73+
74+
- Transformer-heavy architectures: e.g. attention blocks, transformer encoders, ViTs, or hybrid CNN+transformer models.
75+
- *Example:* an audio event detection transformer that must process longer sequences or higher-resolution spectrograms.
76+
- High-resolution or multi-branch networks: models with high input dimensionality or multiple processing paths that saturate NPU throughput.
77+
- *Example:* 512×512 semantic segmentation or multi-object detection.
78+
- Dense post-processing or large fully connected layers: cases where U55/U65 memory limits or MAC bandwidth previously restricted performance.
79+
- *Example:* large MLP heads or transformer token mixers.
80+
- Multi-modal pipelines: combining multiple sensor inputs (e.g. image + IMU + audio) where the NPU must maintain concurrency or shared intermediate representations.
81+
82+
The Ethos-U85 is ideal for projects where model performance is constrained by attention layers, large activations, or operator types that previously required fallback to the CPU. Use the Ethos-U85 to eliminate those fallbacks and achieve full-NPU execution of advanced topologies.
83+
84+
---
85+
86+
## What will you use?
87+
You should be familiar with, or willing to learn about:
88+
- Programming: Python, C/C++
89+
- ExecuTorch or TensorFlow Lite (Micro/LiteRT)
90+
- Techniques for optimising AI models for the edge (quantization, pruning, etc.)
91+
- Optimization Tools:
92+
- TOSA Model Explorer
93+
- .tflite to .tosa converter (if using Tensorflow rather than ExecuTorch)
94+
- Vela compiler for Ethos-U
95+
- Bare-metal or RTOS (e.g., Zephyr)
96+
97+
---
98+
99+
## Resources from Arm and our partners
100+
- Arm Developer: [Edge AI](https://developer.arm.com/edge-ai)
101+
- Learning Path: [Navigating Machine Learning with Ethos-U processors](https://learn.arm.com/learning-paths/microcontrollers/nav-mlek/)
102+
- Repository: [AI on Arm course](https://github.com/arm-university/AI-on-Arm)
103+
- Example Board: [Alif Ensemble DevKit E8](https://www.keil.arm.com/boards/alif-semiconductor-devkit-e8-gen-1-2558a7b/features/)
104+
- Documentation: [TOSA Specification](https://www.mlplatform.org/tosa/), [TOSA Model Explorer](https://github.com/arm/tosa-adapter-model-explorer), and [TOSA Reference Model](https://gitlab.arm.com/tosa/tosa-reference-model)
105+
- PyTorch Blog: [ExecuTorch support for Ethos-U85](https://pytorch.org/blog/pt-executorch-ethos-u85/)
106+
---
107+
108+
## Support Level
109+
110+
This project is designed to be self-serve but comes with opportunity of some community support from Arm Ambassadors, who are part of the Arm Developer program. If you are not already part of our program, [click here to join](https://www.arm.com/resources/developer-program?#register).
111+
112+
## Benefits
113+
114+
Standout project contributions will result in digital badges for CV building, recognised by Arm Talent Acquisition. We are currently discussing with national agencies the potential for funding streams for Arm Developer Labs projects, which would flow to you, not us.
115+
116+
117+
To receive the benefits, you must show us your project through our [online form](https://forms.office.com/e/VZnJQLeRhD). Please do not include any confidential information in your contribution. Additionally if you are affiliated with an academic institution, please ensure you have the right to share your material.

‎Projects/Projects/NGP.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
---
2+
title: "Game development using Arm Neural Graphics with Unreal Engine"
3+
description: "Build a playable Unreal Engine 5 game demo that utilises Arm’s Neural Graphics SDK UE plugin for features such as Neural Super Sampling (NSS). Showcase near-identical image quality at lower resolution by driving neural rendering directly in the graphics pipeline."
4+
subjects:
5+
- "ML"
6+
- "Gaming"
7+
- "Libraries"
8+
- "Graphics"
9+
requires-team:
10+
- "No"
11+
platform:
12+
- "Mobile, Graphics, and Gaming"
13+
- "Laptops and Desktops"
14+
- "AI"
15+
sw-hw:
16+
- "Software"
17+
support-level:
18+
- "Self-Service"
19+
- "Arm Ambassador Support"
20+
publication-date: 2025-11-27
21+
license:
22+
status:
23+
- "Published"
24+
donation:
25+
---
26+
27+
![educate_on_arm](../../images/Educate_on_Arm_banner.png)
28+
29+
## Description
30+
31+
### Why is this important?
32+
33+
Arm neural technology is an industry first, adding dedicated neural accelerators to Arm GPUs, bringing PC-quality, AI powered graphics to mobile for the first time – and laying the foundation for future on-device AI innovation.
34+
35+
Developers can start building now with the industry’s first open development kit for neural graphics with an Unreal Engine plugin, emulators, and open models on GitHub and Hugging Face.
36+
37+
[Arm Neural Technology Announcement](https://newsroom.arm.com/news/arm-announces-arm-neural-technology)
38+
39+
Neural Super Sampling (NSS) is Arm’s mobile-optimized AI-driven graphics upscaler that improves image quality while lowering resolution. It builds on a prior Arm solution: Accuracy Super Resolution (ASR). It is supported by an Unreal Engine plugin, streamlining its use as part of a typical industry games development process.
40+
41+
Future SDK support will be provided for Neural Frame Rate Upscaling (NFRU) - so feel free to extend this project using NFRU when released.
42+
43+
### Project Summary
44+
45+
Create a small game scene utilising the Arm Neural Graphics UE plugin to demonstrate:
46+
- **Near-identical visuals at lower resolution** (render low → upscale with NSS)
47+
48+
Document your progress and findings and consider alternative applications of the neural technology within games development.
49+
50+
Attempt different environments and objects. For example:
51+
52+
- Daytime vs night
53+
- Urban city, jungle forest, ocean floor, alien planet, building interiors
54+
- Complex lighting and shadows
55+
- NPCs with detailed clothing, faces, hair. Include animations.
56+
57+
Make your scenes dynamic with particle effects, shadows, physics and motion.
58+
59+
---
60+
61+
## Pre-requisites
62+
- Laptop/PC/Mobile for Android Unreal Engine game development
63+
- Willingness to learn about games development and graphics, and the increasing use of AI in these fields.
64+
65+
---
66+
67+
## Resources from Arm and partners
68+
- Get Started Blog: [Start experimenting with NSS today](https://developer.arm.com/community/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/how-to-access-arm-neural-super-sampling)
69+
- Deep Dive Blog: [How NSS works](https://developer.arm.com/community/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/how-arm-neural-super-sampling-works)
70+
- Arm Developer: [Neural Graphics Development Kit](https://developer.arm.com/mobile-graphics-and-gaming/neural-graphics)
71+
- Learning Path: [Fine-tuning neural graphics models with Model Gym](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/model-training-gym/)
72+
- Learning Path: [Neural Super Sampling in Unreal Engine](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/nss-unreal/)
73+
- Learning Path: [Getting started with Arm Accuracy Super Resolution (Arm ASR)](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/)
74+
- Unreal Engine Intro by Epic Games: [Understanding the basics](https://dev.epicgames.com/documentation/en-us/unreal-engine/understanding-the-basics-of-unreal-engine)
75+
- Repo: [Arm Neural Graphics SDK](https://github.com/arm/neural-graphics-sdk-for-game-engines)
76+
- Repo: [Arm Neural Graphics Model Gym](https://github.com/arm/neural-graphics-model-gym)
77+
- Documentation: [Arm Neural Graphics SDK for Game Engines Developer guide](https://developer.arm.com/documentation/111167/latest/)
78+
79+
---
80+
81+
## Support Level
82+
83+
This project is designed to be self-serve but comes with opportunity of some community support from Arm Ambassadors, who are part of the Arm Developer program. If you are not already part of our program, [click here to join](https://www.arm.com/resources/developer-program?#register).
84+
85+
## Benefits
86+
87+
Standout project contributions will result in digital badges for CV building, recognised by Arm Talent Acquisition. We are currently discussing with national agencies the potential for funding streams for Arm Developer Labs projects, which would flow to you, not us.
88+
89+
90+
To receive the benefits, you must show us your project through our [online form](https://forms.office.com/e/VZnJQLeRhD). Please do not include any confidential information in your contribution. Additionally if you are affiliated with an academic institution, please ensure you have the right to share your material.

0 commit comments

Comments
 (0)