Merge pull request #2219 from annietllnd/neural-graphics

Add LPs on neural graphics

Author: pareenaverma

content/learning-paths/mobile-graphics-and-gaming/nss-unreal/1-install-plugin.md

@@ -0,0 +1,45 @@
+---
+title: Introduction to neural graphics and Neural Super Sampling (NSS)
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## What is the Neural Graphics Development Kit?
+
+The Neural Graphics Development Kit empowers game developers to build immersive mobile gaming experiences using a neural accelerator for post-processing effects like upscaling. By combining Unreal Engine and the ML extensions for Vulkan, these tools allow you to integrate and evaluate AI-based upscaling technologies like Neural Super Sampling (NSS). This Learning Path walks you through the setup and execution of NSS for Unreal Engine.
+
+## What is Neural Super Sampling?
+
+NSS is an upscaling technology from Arm, purpose-built for real-time performance and power efficiency on mobile and embedded platforms.
+
+It uses a compact neural network to:
+- Upscale low-resolution frames into high-resolution visuals
+- Incorporate temporal data such as motion vectors, depth, and feedback
+- Reduce bandwidth usage and GPU load
+
+Powered by the ML extensions for Vulkan, this new technology delivers smooth, crisp image quality, optimized for **mobile-class hardware** with a **Neural Accelerator** (NX). You’ll be able to render frames at a lower resolution and then upscale them using the technology, which helps you achieve higher frame rates without compromising the visual experience. This is especially useful on mobile, handheld, or thermally limited platforms, where battery life and thermal headroom are critical. It can also deliver improved image quality compared to other upsampling techniques, like spatio-temporal implementations.
+
+Under the hood, Neural Super Sampling for Unreal Engine (NSS for UE) runs its neural inference through Vulkan using **ML extensions for Vulkan**, which bring machine learning workloads into the graphics pipeline. The Development Kit includes **emulation layers** that simulate the behavior of the extensions on Vulkan compute capable GPUs. These layers allow you to test and iterate without requiring access to NX hardware.
+
+## Neural Upscaling in Unreal Engine
+
+With these resources, you can seamlessly integrate NSS into any Unreal Engine project. The setup is designed to work with Vulkan as your rendering backend, and you don’t need to overhaul your workflow - just plug it in and start leveraging ML-powered upscaling right away. The technology is available as a source-code implementation that you will build with Visual Studio.
+
+## Download required artifacts
+
+Before you begin, download the required plugins and dependencies. These two repositories contain everything you need to set up NSS for Unreal Engine, including the VGF model file, and the ML Emulations Layers for Vulkan.
+
+### 1. Download the NSS plugin
+
+[**Neural Super Sampling Unreal Engine Plugin** → GitHub Repository](https://github.com/arm/neural-graphics-for-unreal)
+
+Download the latest release package and extract it on your Windows machine.
+
+### 2. Download the runtime for ML Extensions for Vulkan
+[**Unreal NNE Runtime RDG for ML Extensions for Vulkan** → GitHub Repository](https://github.com/arm/ml-extensions-for-vulkan-unreal-plugin).
+
+Download and extract the release package on your Windows machine.
+
+Once you’ve extracted both repositories, proceed to the next section to set up your development environment and enable the NSS plugin.

content/learning-paths/mobile-graphics-and-gaming/nss-unreal/2-emulation-layer.md

@@ -0,0 +1,81 @@
+---
+title: Setting up the emulation layers
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Install dependencies
+
+To run NSS in your Unreal Engine project, install and configure the following:
+
+- **Vulkan SDK**: Required for development of applications that use Vulkan, and to enable the Vulkan Configurator. The latter sets up the emulation layers used for running ML extensions for Vulkan workloads.
+- **ML Emulation Layer for Vulkan**: These layers allows neural inference to run in emulation through Vulkan’s compute backend. They are activated by Vulkan Configurator to run with the Unreal Engine plugin. The layers are included in the `NNERuntimeRDGMLExtensionsForVulkan` zip you downloaded in a previous step. The Vulkan layer configuration activates the ML Emulation Layer for Vulkan, which implements the ML extensions for Vulkan.
+- **NSS for Unreal Engine plugins**: These include `NSS` (the inference and model interface) and `NNERuntimeRDGMLExtensionsForVulkan` (which connects Unreal’s Render Dependency Graph to the ML extensions for Vulkan).
+
+These components allow you to run NSS in Unreal Engine, using ML emulation layers for Vulkan for development and testing.
+
+## Install Vulkan Software Development Kit
+
+Go to the [Vulkan SDK landing page](https://vulkan.lunarg.com/sdk/home) and download the SDK Installer for Windows. After you have run the installer, you can move on to the next step.
+
+## Configure Vulkan Layers
+
+Vulkan Configurator is a program that will run the emulation layers in the background when you want to utilize them with Unreal Engine.
+
+To emulate the ML extensions for Vulkan:
+1. Launch the **Vulkan Configurator** (bundled with the Vulkan SDK) from the Windows **Start** menu.
+2. In the **Apply a Vulkan Loader Configuration** list, right-click and choose **Create a new Configuration**. You can give the new configuration any name, for example `NSS`.
+3. Navigate to the **Vulkan Layers Location** tab.
+4. Append a user-defined path pointing to the emulation layers you downloaded in the previous section:
+   ```
+   <download-path>/NNERuntimeRDGMLExtensionsForVulkan/MLEmulationLayerForVulkan
+   ```
+![Add user-defined Vulkan layers path in Vulkan Configurator#center](./images/load_layers.png "Figure 1: Add Vulkan layer path.")
+
+5. Ensure the Graph layer is listed *above* the Tensor layer, and that you've set up the configuration scope as shown in the image.
+
+![Layer configuration showing Graph above Tensor#center](./images/verify_layers.png "Figure 2: Verify layer ordering and scope.")
+
+
+{{% notice %}}
+Keep the Vulkan Configurator running to enable the emulation layers during engine execution.
+{{% /notice %}}
+
+## Enable NSS for Unreal Engine
+
+1. Open Unreal Engine and create a new **Third Person** template project using the **C++** option.
+
+![Unreal Engine project selection screen showing C++ Third Person template#center](./images/unreal_startup.png "Figure 3: Create a new C++ project in Unreal Engine.")
+
+2. Open the project in **Visual Studio**. Build it from source through **Build** > **Build Solution** or with `Ctrl+Shift+B`.
+
+After the build is finished, open your project in Unreal Engine.
+
+## Change Unreal’s Rendering Interface to Vulkan
+
+By default, Unreal uses DirectX. Instead, you need to choose Vulkan as the default RHI:
+1. Go to:
+   ```
+   Project Settings > Platform > Windows > Targeted RHIs > Default RHI
+   ```
+2. Select **Vulkan**.
+3. Restart Unreal Engine to apply the change.
+
+![Project Settings with Vulkan selected as Default RHI under Targeted RHIs#center](./images/targeted_rhis.png "Figure 4: Set Vulkan as the default RHI.")
+
+
+## Add and enable the plugins
+
+1. Open your project directory in Windows explorer, and create a new folder called `Plugins`.
+2. Copy the downloaded and extracted `.zips` into the new directory:
+   - `NNERuntimeRDGMLExtensionsForVulkan`
+   - `NSS`
+3. Re-open Unreal Engine. When prompted, confirm plugin integration.
+4. Rebuild your project in Visual Studio from source.
+5. Verify the installation by opening the Plugins view in Unreal Engine, and making sure the checkbox is selected for both `NSS` and `NNERuntimeRDGMLExtensionsForVulkan` as shown. Restart Unreal Engine if prompted.
+
+![Unreal Engine plugins window showing NSS and NNERuntimeRDGMLExtensionsForVulkan enabled#center](./images/verify_plugin_enabled.png "Figure 5: Verify plugin installation in Unreal Engine.")
+
+With the emulation layers and plugins configured, you're ready to run Neural Super Sampling in Unreal Engine. Continue to the next section to test the integration.

content/learning-paths/mobile-graphics-and-gaming/nss-unreal/3-run-example.md

@@ -0,0 +1,42 @@
+---
+title: Run the example
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Start the level and verify NSS
+
+Press the green **Play** button to start the level. To verify NSS is running, you can run this command in Unreal:
+   ```
+   ShowFlag.VisualizeTemporalUpscaler 1
+   ```
+You’ll see **NSS** listed in the rendering summary.
+
+{{% notice %}}
+In **Project Settings > Plugins > Neural Super Sampling**, you can view and configure the active neural network model being used.
+{{% /notice %}}
+
+Run `ShowFlag.VisualizeTemporalUpscaler 0` to disable the overview. To visualize the NSS model output in real-time, run the following command:
+   ```
+   r.NSS.Debug 2
+   ```
+
+This will add real-time views showing the model’s processed outputs, such as predicted filter coefficients and feedback, as below. In the [Wrapping up section](./6-wrapping-up.md), you will find links to learn more about what the debug outputs mean.
+
+![Debug view of Neural Super Sampling model output in Unreal Engine#center](./images/nss_debug.png "Figure 6: Visualize NSS model debug output in real time.")
+
+## NSS model on Hugging Face
+
+The model that powers NSS is published on Hugging Face in the [VGF format](https://github.com/arm/ai-ml-sdk-vgf-library). This format is optimized for inference via ML extensions for Vulkan.
+
+Visit the [NSS model page on Hugging Face](https://huggingface.co/Arm/neural-super-sampling/)
+
+On this landing page, you can read more about the model, and learn how to run a test case - a _scenario_ - using the ML SDK for Vulkan.
+
+## Result
+
+You now have Neural Super Sampling integrated and running inside Unreal Engine. This setup provides a real-time testbed for neural upscaling.
+
+Proceed to the next section to debug your frames using RenderDoc, or move on to the final section to explore more resources on the technology behind NSS.

content/learning-paths/mobile-graphics-and-gaming/nss-unreal/5-renderdoc.md

@@ -0,0 +1,70 @@
+---
+title:  Using RenderDoc for Debugging and Analysis
+weight: 6
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Why use RenderDoc with Neural Super Sampling?
+
+As you integrate neural upscaling techniques into your game, visual debugging and performance profiling become essential. RenderDoc is a powerful frame capture and analysis tool that allows you to step through a frame, inspect Vulkan API calls, view shader inputs and outputs, and understand the state of resources. Arm has released some additional features, which are captured in [RenderDoc for Arm GPUs](https://developer.arm.com/Tools%20and%20Software/RenderDoc%20for%20Arm%20GPUs).
+
+You might want to use RenderDoc when:
+
+- You see unexpected visual output or want to step through the frame rendering process.
+- You need to analyze the sequence of Vulkan API calls made by the engine.
+- You’re inspecting memory usage or the state of specific GPU resources.
+- You want to validate your data graph pipeline execution or identify synchronization issues.
+
+## Install Arm Performance Studio
+
+To access RenderDoc for Arm GPUs containing the added features with ML extensions for Vulkan, you should install Arm Performance Studio. Download it from the [Arm Performance Studio Downloads](https://developer.arm.com/Tools%20and%20Software/Arm%20Performance%20Studio#Downloads). The minimum version to use is `2025.4`
+
+Refer to the [Arm Performance Studio install guide](./install-guides/ams) to set it up.
+
+Upon a finished installation, you can find the installed version of RenderDoc for Arm GPUs using the Windows **Start** menu.
+
+## Use in Unreal Engine
+
+### 1. Configure the executable path
+
+To enable integration with Unreal Engine:
+
+1. Open your Unreal Engine project.
+2. Go to **Edit > Project Settings > Plugins > RenderDoc**.
+3. Under **Path to RenderDoc executable**, enter the full path to the directory where the `qrenderdoc.exe` binary is located.
+4. Restart Unreal Engine for the setting to take effect.
+
+![RenderDoc plugin path setup in Unreal Engine#center](./images/renderdoc_plugin_ue.png "Figure 7: Set the RenderDoc executable path in Unreal Engine plugin settings.")
+
+
+## 2. Ways to capture
+
+### Option 1: Attach to the Running Editor
+
+1. Launch RenderDoc for Arm GPUs separately.
+2. Go to **File > Attach to Running Instance**.
+3. A list of running Vulkan-enabled applications will appear. Select the hostname that corresponds to the UE Editor session (with UI) or use Standalone Running App (see image below).
+4. Click **Connect to App**.
+5. Click **Capture Frame Immediately** or set up the capture settings otherwise.
+
+### Option 2: Use plugin inside Unreal Engine
+1. Open your project and scene where you want to perform a capture.
+2. Click the **RenderDoc Capture** button in the Level Viewport (see image below).
+
+![RenderDoc capture button in Unreal Engine Level Viewport, or Attach to Running Instance #center](./images/renderdoc.png "Figure 8: Two options to capture frames using RenderDoc with Unreal Engine.")
+
+## 3. Capture a Frame
+
+1. Return to Unreal Engine and **Play in Editor** to launch your game level.
+2. In RenderDoc for Arm GPUs, click **Capture Frame Now** (camera icon) or press `F12` while the UE window is focused.
+3. Once captured, double-click the frame in RenderDoc to open a detailed breakdown of the GPU workload.
+
+You can now:
+
+- Step through draw calls and dispatches.
+- Inspect bound resources, descriptor sets, and shaders.
+- Explore the execution of your data graph pipeline frame-by-frame.
+
+If you want to learn more about RenderDoc for Arm GPUs, you can check out the [Debug With RenderDoc User Guide](https://developer.arm.com/documentation/109669/latest).

content/learning-paths/mobile-graphics-and-gaming/nss-unreal/6-wrapping-up.md

@@ -0,0 +1,25 @@
+---
+title:  Wrapping up
+weight: 7
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+With the NSS for UE plugin, you’re set up to explore real-time neural graphics with Neural Super Sampling. This toolchain gives you direct access to state-of-the-art upscaling powered by machine learning.
+
+You’ve covered:
+- Understanding the role of **ML Extensions for Vulkan** and how emulation layers let you run everything without needing dedicated ML hardware
+- Installing the **Vulkan SDK** and enabling ML Emulation Layer for Vulkan using Vulkan Configurator
+- Setting up the **NSS for Unreal Engine** plugins, and visualizing the model output
+- Inspecting the **NSS model** in VGF on Hugging Face
+
+This ecosystem is built for developers who want to push boundaries - whether on flagship mobile SoCs or desktop dev kits. NSS is designed to give you better image quality without the complexity of building custom ML infrastructure.
+
+To learn more about the different aspects in this Learning Path, check out the following resources:
+- [Neural Graphics Development Kit landing page](https://developer.arm.com/mobile-graphics-and-gaming/neural-graphics-for-mobile)
+- [NSS Use Case Guide](https://developer.arm.com/documentation/111009/latest/)
+- [Debugging NSS content with RenderDoc](https://developer.arm.com/documentation/109669/latest)
+- [Learning Path: Get started with neural graphics using ML Extensions for Vulkan](/learning-paths/mobile-graphics-and-gaming/vulkan-ml-sample)
+
+Happy building - and welcome to the future of neural upscaling in Unreal!

content/learning-paths/mobile-graphics-and-gaming/nss-unreal/_index.md

@@ -0,0 +1,63 @@
+---
+title: Neural Super Sampling in Unreal Engine
+
+minutes_to_complete: 30
+
+who_is_this_for: This is an introductory topic for developers experimenting with neural graphics using Unreal Engine® and ML Extensions for Vulkan®.
+
+
+learning_objectives:
+    - Understand how Arm enables neural graphics for game development
+    - Configure ML extensions for Vulkan emulation
+    - Enable Neural Super Sampling (NSS) in Unreal Engine
+    - Run and visualize real-time upscaling with NSS
+
+
+prerequisites:
+    - Windows 11
+    - Unreal Engine 5.5 (Templates and Feature Pack enabled)
+    - Visual Studio 2022 (with Desktop Development with C++ and .NET desktop build tools)
+
+
+author: Annie Tallund
+
+### Tags
+skilllevels: Introductory
+subjects: ML
+armips:
+    - Mali
+tools_software_languages:
+    - Unreal Engine
+    - Vulkan SDK
+    - Visual Studio
+operatingsystems:
+    - Windows
+
+
+
+further_reading:
+    - resource:
+        title: Neural Graphics Development Kit
+        link: https://developer.arm.com/mobile-graphics-and-gaming/neural-graphics-for-mobile
+        type: website
+    - resource:
+        title: NSS Use Case Guide
+        link: https://developer.arm.com/documentation/111009/latest/
+        type: documentation
+    - resource:
+        title: RenderDoc for Arm GPUs
+        link: https://developer.arm.com/Tools%20and%20Software/RenderDoc%20for%20Arm%20GPUs
+        type: documentation
+    - resource:
+        title: How Arm Neural Super Sampling works
+        link: https://community.arm.com/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/how-arm-neural-super-sampling-works
+        type: blog
+
+
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1                       # _index.md always has weight of 1 to order correctly
+layout: "learningpathall"       # All files under learning paths have this same wrapper
+learning_path_main_page: "yes"  # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
+---

content/learning-paths/mobile-graphics-and-gaming/nss-unreal/_next-steps.md

@@ -0,0 +1,8 @@
+---
+# ================================================================================
+#       FIXED, DO NOT MODIFY THIS FILE
+# ================================================================================
+weight: 21                  # Set to always be larger than the content in this path to be at the end of the navigation.
+title: "Next Steps"         # Always the same, html page title.
+layout: "learningpathall"   # All files under learning paths have this same wrapper for Hugo processing.
+---