New Projects

New Projects - Edge AI and Mobile

Author: matt-cossins

Projects/Projects/Edge-AI-On-Mobile.md

@@ -0,0 +1,76 @@
+---
+title: "SME2 on vivo X300: Mobile Edge AI Projects for multi-modal inference, built on Arm Lumex"
+description: "Leverage the latest SME2 (Scalable Matrix Extension 2) available on the newest vivo X300 smartphones (built on Arm Lumex CSS) for advanced image/video, audio and text processing edge AI. Explore how SME2, via KleidiAI, enables larger matrix workloads, higher throughput, and novel applications on device without cloud connectivity required."
+subjects:
+  - "ML"
+  - "Performance and Architecture"
+  - "Libraries"
+requires-team:
+  - "No"
+platform:
+  - "Mobile, Graphics, and Gaming"
+  - "AI"
+  - "IoT"
+sw-hw:
+  - "Software"
+support-level:
+  - "Self-Service"
+  - "Arm Ambassador Support"
+publication-date: 2025-11-27
+license:
+status:
+  - "Published"
+---
+
+## Description
+
+### Why is this important?
+
+SME2 (Scalable Matrix Extension 2) is the latest CPU extension on Arm Lumex CSS. Designed to accelerate matrix-oriented compute workloads directly on device, SME2 improves AI/ML performance. This is by accelerating models that rely on operations like matrix multiplication, common in transformers, convolutional neural networks (CNNs), and large language models (LLMs). Via KleidiAI, SME2 is seamlessly integrated into frameworks such as ExecuTorch, LiteRT, ONNX Runtime so it is automatically leveraged for applications depending on whether SME2 is present on the host device.
+
+[SME2](https://www.arm.com/technologies/sme2)
+
+The vivo X300 is built on Arm Lumex. SME2 now enables AI compute that previously was too heavy or inaccessible on mobile. Developers can now utilise these advancements to deliver advanced applications on-device, reducing latency, increasing data privacy, and unlocking novel use-cases.
+
+[vivo X300, built on Arm Lumex](https://www.arm.com/company/success-library/vivo-x300-smartphones)
+
+### Project Summary
+
+Select a **mobile edge AI application** that benefits from large matrix operations, multi-modal fusion, or transformer-based processing enabled by SME2. Build and optimize a proof-of-concept application on a vivo X300 phone or other device supporting SME2.
+
+Example project areas:  
+    - Real-time video semantic segmentation (e.g., background removal + AR compositing)  
+    - Live object detection + natural-language description (text summary of what the camera sees)  
+    - Multi-sensor fusion (camera + IMU + microphone) for gesture + voice recognition  
+    - On-device lightweight LLM or encoder-only transformer processing for mobile assistants  
+
+Identify a model architecture that maps to wide matrix operations (e.g., ViT, MLP-Mixer, multi-branch CNN with large FC layers). Utilise a mobile-friendly framework (e.g., ExecuTorch, LiteRT, ONNX Runtime, MediaPipe) to leverage SME2 optimizations. Optimize quantization, memory layout, and verify that the large matrix multiplications get scheduled efficiently on the SME2-enabled CPU. Build a mobile app (Android) that executes the model and utilises it for a compelling use-case.
+
+Utilise the resources and learning paths below and create an exciting and challenging application. Optionally, you could also compare performance vs a reference phone without SME2.
+
+---
+
+## Resources from Arm and our partners
+
+- Arm Developer: [Launchpad - Mobile AI](https://developer.arm.com/mobile-graphics-and-gaming/ai-mobile)
+- Learning Path: [Mobile AI/ML Performance Profiling](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/profiling-ml-on-arm/)
+- Learning Path: [Build an Android chat app with Llama, KleidiAI, ExecuTorch, and XNNPACK](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/build-llama3-chat-android-app-using-executorch-and-xnnpack/)
+- Learning Path: [Vision LLM Inference on Android with KleidiAI](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/)
+- Learning Path: [Build a Hands-Free Selfie Android Application with MediaPipe](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/build-android-selfie-app-using-mediapipe-multimodality/)
+- Repository: [AI on Arm course](https://github.com/arm-university/AI-on-Arm)
+- Arm / Cambridge University edX course: [AI at the Edge on Arm (Mobile)](https://www.edx.org/learn/computer-science/arm-education-ai-at-the-edge-on-arm)
+
+---
+
+## Support Level
+
+This project is designed to be self-serve but comes with opportunity of some community support from Arm Ambassadors, who are part of the Arm Developer program. If you are not already part of our program, [click here to join](https://www.arm.com/resources/developer-program?#register).
+
+## Benefits 
+
+Standout project contributions will result in digital badges for CV building, recognised by Arm Talent Acquisition. We are currently discussing with national agencies the potential for funding streams for Arm Developer Labs projects, which would flow to you, not us.
+
+
+To receive the benefits, you must show us your project through our [online form](https://forms.office.com/e/VZnJQLeRhD). Please do not include any confidential information in your contribution. Additionally if you are affiliated with an academic institution, please ensure you have the right to share your material.
+
+---

Projects/Projects/Edge-AI-with-Ethos-U85-NPU.md

@@ -0,0 +1,80 @@
+---
+title: "Edge AI with NPU: always-on-AI with ExecuTorch on Cortex-M55 + Ethos-U85 → Cortex-A"
+description: "The vision of Edge AI compute is to embed low-power intelligent sensing, perception, and decision systems everywhere. A low-power always-on-AI island continuously monitors sensory inputs to detect triggers. When a trigger is detected, it wakes up a more capable processor to carry out high-value inference, interaction, or control tasks."
+subjects:
+    - "ML"
+    - "Performance and Architecture"
+    - "Embedded Linux"
+    - "RTOS Fundamentals"
+requires-team:
+    - "No"
+platform:
+    - "IoT"
+    - "Embedded and Microcontrollers"
+    - "AI"
+sw-hw:
+    - "Software"
+    - "Hardware"
+support-level: 
+    - "Self-Service"
+    - "Arm Ambassador Support"
+publication-date: 2025-11-27
+license:
+status:
+    - "Published" 
+donation: 
+---
+
+![educate_on_arm](../../images/Educate_on_Arm_banner.png)
+
+
+## Description
+
+**Why is this important?**
+
+The vision of Edge AI compute is to embed intelligent low-power sensing, perception, and decision systems everywhere (in homes, wearables, infrastructure) so devices can react to subtle cues, adapt to context, and wake higher-power systems only when needed. Rather than sending everything to the cloud or running full-scale models continuously, this Edge AI system operates as a layered hierarchy:  
+- A low-power always-on-AI model continuously monitors sensory inputs (audio, motion, video) to detect triggers or anomalies.  
+- When a trigger is detected, it wakes up a more capable processor (e.g. Cortex-A running a rich OS such as Linux) to carry out further tasks. This could be high-value inference, interaction, or control tasks. It could also involve connecting to other IoT devices or to a Neoverse cloud instance.
+
+This architecture is key to bridging the gap between battery-constrained devices and rich AI services, making systems smarter, more efficient, and responsive without draining resources.
+
+**Project Summary**
+
+Using equipment such as the Alif Ensemble development kit (e.g. E6/E8, which includes Cortex-A, Cortex-M55, and Ethos-U85 cores - or E4 (M55+U85) + Raspberry Pi for Cortex-A), and the ExecuTorch framework, build an Edge AI prototype that implements:  
+
+1. A “wake-up” path: deploy a TOSA-compliant optimized model on the Cortex-M55 + Ethos-U85 pair to continuously monitor sensory signals (audio, motion, video) for wake-word, anomalies, or triggers.  
+2. A subsequent workload path: when a trigger is detected, activate a Cortex-A core to perform more complex tasks, e.g. use an LLM optimised for CPU inference, connect to and manage other IoT devices, or connect to a Neoverse cloud instance for heavier inference.  
+3. Evaluation and documentation: measure accuracy, latency, power consumption, robustness, and compare trade-offs between modalities (audio, video, motion). Demonstrate an end-to-end use case of your choice (e.g. smart assistant, anomaly alert system, gesture control, environment monitoring).  
+
+*Note that the Cortex-A32 included on the Alif DevKits will not be suitable for LLM inference. If using the onboard core for the project, target cloud/IoT connectivity. For LLM inference, consider connecting a Raspberry Pi 5 or similar.*
+
+Example: Use a microphone input to detect “Hey Arm”. After wake-up, launch an optimised LLM on Raspberry Pi Cortex-A to answer questions or control local devices.
+
+You are free to mix and match sensors, modalities, and tasks — as long as the core architecture (wake-on M55/U85, main task on A) is preserved.
+
+Many of these DevKits come with additional Ethos-U55 NPUs onboard - feel free to be creative and distribute different tasks across the different NPUs - what use-cases and applications can you achieve?
+
+## What will you use?
+You should either be familiar with, or willing to learn about, the following:
+-	Programming: Python, C++, Embedded C
+-	ExecuTorch, plus knowledge of model quantization, pruning, conversion. Use of the Vela compiler and TOSA.
+-	Edge/Embedded development: bare-metal or RTOS (e.g. Zephyr), and embedded Linux (e.g. Yocto) or Raspberry Pi OS
+
+
+## Resources from Arm and our partners
+- Arm Developer: [Edge AI](https://developer.arm.com/edge-ai)
+- Learning Path: [Navigating Machine Learning with Ethos-U processors](https://learn.arm.com/learning-paths/microcontrollers/nav-mlek/)    
+- Repository: [AI on Arm course](https://github.com/arm-university/AI-on-Arm)  
+- Example Board: [Alif Ensemble DevKit E8](https://www.keil.arm.com/boards/alif-semiconductor-devkit-e8-gen-1-2558a7b/features/)  
+- PyTorch Blog: [ExecuTorch support for Ethos-U85](https://pytorch.org/blog/pt-executorch-ethos-u85/)
+
+## Support Level
+
+This project is designed to be self-serve but comes with opportunity of some community support from Arm Ambassadors, who are part of the Arm Developer program. If you are not already part of our program, [click here to join](https://www.arm.com/resources/developer-program?#register).
+
+## Benefits 
+
+Standout project contributions will result in digital badges for CV building, recognised by Arm Talent Acquisition. We are currently discussing with national agencies the potential for funding streams for Arm Developer Labs projects, which would flow to you, not us.
+
+
+To receive the benefits, you must show us your project through our [online form](https://forms.office.com/e/VZnJQLeRhD). Please do not include any confidential information in your contribution. Additionally if you are affiliated with an academic institution, please ensure you have the right to share your material.

Projects/Projects/Ethos-U85-NPU-Applications.md

@@ -0,0 +1,117 @@
+---
+title: "Ethos-U85 NPU Applications with TOSA Model Explorer: Exploring Next-Gen Edge AI Inference"
+description: "Push the limits of Edge AI by deploying the heaviest inference applications possible on Ethos-U85. Students will explore transformer-based and TOSA-optimized workloads that demonstrate performance levels on the next-gen of Ethos NPUs."
+subjects:
+  - "ML"
+  - "Performance and Architecture"
+requires-team:
+  - "No"
+platform:
+  - "IoT"
+  - "Embedded and Microcontrollers"
+  - "AI"
+sw-hw:
+  - "Software"
+  - "Hardware"
+support-level:
+  - "Self-Service"
+  - "Arm Ambassador Support"
+publication-date: 2025-11-27
+license:
+status:
+  - "Published"
+donation:
+---
+
+![educate_on_arm](../../images/Educate_on_Arm_banner.png)
+
+## Description
+
+**Why is this important?**
+
+The Arm Ethos-U85 NPU represents a major leap in bringing *heavy inference* to constrained embedded systems. With its full transformer operator support, expanded MAC throughput, and native TOSA compatibility, the Ethos-U85 enables developers to deploy models and workloads that were previously too intensive for MCU-class devices.
+
+This project challenges you to explore the boundaries of what’s possible on Ethos-U85. The goal is to demonstrate inference performance and model complexity that is now achievable due to the architectural improvements and transformer acceleration capabilities of the Ethos-U85.
+
+[Ethos-U85 Launch](https://newsroom.arm.com/blog/ethos-u85)  
+
+**Project Summary**
+
+Using hardware such as the Alif Ensemble E4/E6/E8 DevKits (all include Ethos-U85) or a comparable platform or Arm Fixed Virtual Platform Corstone-320, your task is to design and benchmark an advanced edge inference application that exploits the Ethos-U85’s compute and transformer capabilities.
+
+Your project should include:
+
+1. Model Deployment and Optimization  
+   Select a computationally intensive model — ideally transformer-based or multi-branch convolutional — and deploy it on the Ethos-U85 using:
+   - The TOSA Model Explorer extension to inspect and adapt unsupported or experimental models for TOSA compliance.  
+   - The Vela compiler for optimization.  
+
+   These tools can be used to:
+   - Convert and visualize model graphs in TOSA format.  
+   - Identify unsupported operators.  
+   - Modify or substitute layers for compatibility using the Flatbuffers schema before re-exporting.  
+   - Run Vela for optimized compilation targeting Ethos-U85.
+
+2. Application Demonstration 
+   Implement a working example that highlights the Ethos-U85’s strengths in real-world inference. Possible categories include:  
+   - Transformers on Edge: lightweight BERT, ViT, or audio transformers (e.g. speech or sound event classification).  
+   - High-resolution Vision: semantic segmentation, object detection on large input sizes, or multi-head perception networks.  
+   - Multi-modal Fusion: combining audio, image, or sensor streams for contextual understanding.  
+
+3. Analysis and Benchmarking 
+   Report quantitative results on:
+   - Inference latency, throughput (FPS or tokens/s), and memory footprint.  
+   - Power efficiency under load (optional).  
+   - Comparative performance versus Ethos-U55/U65 (use available benchmarks for reference or utilise the other Ethos-U NPUs provided in the Alif DevKits).  
+   - The effect of TOSA optimization — demonstrate measurable improvements from graph conversion and operator fusion.
+
+---
+
+## What kind of projects should you target?
+
+To clearly demonstrate the leap from Ethos-U55/U65 to U85, choose projects that meet at least one of the following criteria:
+
+- Transformer-heavy architectures: e.g. attention blocks, transformer encoders, ViTs, or hybrid CNN+transformer models.  
+  - *Example:* an audio event detection transformer that must process longer sequences or higher-resolution spectrograms.  
+- High-resolution or multi-branch networks: models with high input dimensionality or multiple processing paths that saturate NPU throughput.  
+  - *Example:* 512×512 semantic segmentation or multi-object detection.  
+- Dense post-processing or large fully connected layers: cases where U55/U65 memory limits or MAC bandwidth previously restricted performance.  
+  - *Example:* large MLP heads or transformer token mixers.  
+- Multi-modal pipelines: combining multiple sensor inputs (e.g. image + IMU + audio) where the NPU must maintain concurrency or shared intermediate representations.  
+
+The Ethos-U85 is ideal for projects where model performance is constrained by attention layers, large activations, or operator types that previously required fallback to the CPU. Use the Ethos-U85 to eliminate those fallbacks and achieve full-NPU execution of advanced topologies.
+
+---
+
+## What will you use?
+You should be familiar with, or willing to learn about:
+- Programming: Python, C/C++  
+- ExecuTorch or TensorFlow Lite (Micro/LiteRT)
+- Techniques for optimising AI models for the edge (quantization, pruning, etc.)
+- Optimization Tools: 
+  - TOSA Model Explorer
+  - .tflite to .tosa converter (if using Tensorflow rather than ExecuTorch)
+  - Vela compiler for Ethos-U  
+- Bare-metal or RTOS (e.g., Zephyr)  
+
+---
+
+## Resources from Arm and our partners
+- Arm Developer: [Edge AI](https://developer.arm.com/edge-ai)
+- Learning Path: [Navigating Machine Learning with Ethos-U processors](https://learn.arm.com/learning-paths/microcontrollers/nav-mlek/)    
+- Repository: [AI on Arm course](https://github.com/arm-university/AI-on-Arm)  
+- Example Board: [Alif Ensemble DevKit E8](https://www.keil.arm.com/boards/alif-semiconductor-devkit-e8-gen-1-2558a7b/features/)  
+- Documentation: [TOSA Specification](https://www.mlplatform.org/tosa/), [TOSA Model Explorer](https://github.com/arm/tosa-adapter-model-explorer), and [TOSA Reference Model](https://gitlab.arm.com/tosa/tosa-reference-model)
+- PyTorch Blog: [ExecuTorch support for Ethos-U85](https://pytorch.org/blog/pt-executorch-ethos-u85/)
+---
+
+## Support Level
+
+This project is designed to be self-serve but comes with opportunity of some community support from Arm Ambassadors, who are part of the Arm Developer program. If you are not already part of our program, [click here to join](https://www.arm.com/resources/developer-program?#register).
+
+## Benefits 
+
+Standout project contributions will result in digital badges for CV building, recognised by Arm Talent Acquisition. We are currently discussing with national agencies the potential for funding streams for Arm Developer Labs projects, which would flow to you, not us.
+
+
+To receive the benefits, you must show us your project through our [online form](https://forms.office.com/e/VZnJQLeRhD). Please do not include any confidential information in your contribution. Additionally if you are affiliated with an academic institution, please ensure you have the right to share your material.