Skip to content

Commit f5300ca

Browse files
Merge pull request #2414 from madeline-underwood/nssmodelgym
Nssmodelgym_JA to review
2 parents 7e60bab + 04b2e25 commit f5300ca

File tree

5 files changed

+58
-41
lines changed

5 files changed

+58
-41
lines changed
Lines changed: 22 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,44 @@
11
---
2-
title: Install Model Gym and Explore Neural Graphics Examples
2+
title: Install Model Gym and explore neural graphics examples
33
weight: 2
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9-
## What is Neural Graphics?
9+
## What is neural graphics?
1010

11-
Neural graphics is an intersection of graphics and machine learning. Rather than relying purely on traditional GPU pipelines, neural graphics integrates learned models directly into the rendering stack. The techniques are particularly powerful on mobile devices, where battery life and performance constraints limit traditional compute-heavy rendering approaches. The goal is to deliver high visual fidelity without increasing GPU cost. This is achieved by training and deploying compact neural networks optimized for the device's hardware.
11+
Neural graphics is an intersection of graphics and machine learning. Rather than relying purely on traditional GPU pipelines, neural graphics integrates learned models directly into the rendering stack. These techniques are particularly powerful on mobile devices, where battery life and performance constraints limit traditional compute-heavy rendering approaches. Your goal is to deliver high visual fidelity without increasing GPU cost. You achieve this by training and deploying compact neural networks optimized for your device's hardware.
1212

1313
## How does Arm support neural graphics?
1414

15-
Arm enables neural graphics through the [**Neural Graphics Development Kit**](https://developer.arm.com/mobile-graphics-and-gaming/neural-graphics): a set of open-source tools that let developers train, evaluate, and deploy ML models for graphics workloads.
15+
16+
Arm enables neural graphics through the [**Neural Graphics Development Kit**](https://developer.arm.com/mobile-graphics-and-gaming/neural-graphics): a set of open-source tools that let you train, evaluate, and deploy ML models for graphics workloads.
17+
1618

1719
At its core are the ML Extensions for Vulkan, which bring native ML inference into the GPU pipeline using structured compute graphs. These extensions (`VK_ARM_tensors` and `VK_ARM_data_graph`) allow real-time upscaling and similar effects to run efficiently alongside rendering tasks.
1820

19-
The neural graphics models can be developed using well-known ML frameworks like PyTorch, and exported to deployment using Arm's hardware-aware pipeline. The workflow converts the model to `.vgf` via the TOSA intermediate representation, making it possible to do tailored model development for you game use-case. This Learning Path focuses on **Neural Super Sampling (NSS)** as the use case for training, evaluating, and deploying neural models using a toolkit called the [**Neural Graphics Model Gym**](https://github.com/arm/neural-graphics-model-gym). To learn more about NSS, you can check out the [resources on Hugging Face](https://huggingface.co/Arm/neural-super-sampling). Additonally, Arm has developed a set of Vulkan Samples to get started. Specifically, `.vgf` format is introduced in the `postprocessing_with_vgf` one. The Vulkan Samples and over-all developer resources for neural graphics is covered in the [introductory Learning Path](/learning-paths/mobile-graphics-and-gaming/vulkan-ml-sample).
2021

21-
Starting in 2026, Arm GPUs will feature dedicated neural accelerators, optimized for low-latency inference in graphics workloads. To help developers get started early, Arm provides the ML Emulation Layers for Vulkan that simulate future hardware behavior, so you can build and test models now.
22+
23+
You can develop neural graphics models using well-known ML frameworks like PyTorch, then export them for deployment with Arm's hardware-aware pipeline. The workflow converts your model to `.vgf` using the TOSA intermediate representation, making it possible to tailor model development for your game use case. In this Learning Path, you will focus on **Neural Super Sampling (NSS)** as the primary example for training, evaluating, and deploying neural models using the [**Neural Graphics Model Gym**](https://github.com/arm/neural-graphics-model-gym). To learn more about NSS, see the [resources on Hugging Face](https://huggingface.co/Arm/neural-super-sampling). Arm has also developed a set of Vulkan Samples to help you get started. The `.vgf` format is introduced in the `postprocessing_with_vgf` sample. For a broader overview of neural graphics developer resources, including the Vulkan Samples, see the introductory Learning Path [Get started with neural graphics using ML Extensions for Vulkan](/learning-paths/mobile-graphics-and-gaming/vulkan-ml-sample/).
24+
25+
26+
27+
Starting in 2026, Arm GPUs will feature dedicated neural accelerators, optimized for low-latency inference in graphics workloads. To help you get started early, Arm provides the ML Emulation Layers for Vulkan that simulate future hardware behavior, so you can build and test models now.
2228

2329
## What is the Neural Graphics Model Gym?
2430

31+
2532
The Neural Graphics Model Gym is an open-source toolkit for fine-tuning and exporting neural graphics models. It is designed to streamline the entire model lifecycle for graphics-focused use cases, like NSS.
2633

27-
Model Gym gives you:
34+
With Model Gym, you can:
35+
36+
- Train and evaluate models using a PyTorch-based API
37+
- Export models to `.vgf` using ExecuTorch for real-time use in game development
38+
- Take advantage of quantization-aware training (QAT) and post-training quantization (PTQ) with ExecuTorch
39+
- Use an optional Docker setup for reproducibility
40+
41+
You can choose to work with Python notebooks for rapid experimentation or use the command-line interface for automation. This Learning Path will walk you through the demonstrative notebooks and prepare you to start using the CLI for your own model development.
2842

29-
- A training and evaluation API built on PyTorch
30-
- Model export to .vgf using ExecuTorch for real-time use in game development
31-
- Support for quantization-aware training (QAT) and post-training quantization (PTQ) using ExecuTorch
32-
- Optional Docker setup for reproducibility
3343

34-
The toolkit supports workflows via both Python notebooks (for rapid experimentation) and command-line interface. This Learning Path will walk you through the demonstrative notebooks, and prepare you to start using the CLI for your own model development.
44+
You're now ready to set up your environment and start working with neural graphics models. Keep going!

content/learning-paths/mobile-graphics-and-gaming/model-training-gym/2-devenv.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,11 @@ weight: 3
66
layout: learningpathall
77
---
88

9-
In this section, you will install a few dependencies into your Ubuntu environment. You'll need a working Python 3.10+ environment with some ML and system dependencies. Make sure Python is installed by verifying that the version is >3.10:
9+
## Overview
10+
11+
In this section, you will install a few dependencies into your Ubuntu environment. You'll need a working Python 3.10+ environment with some ML and system dependencies.
12+
13+
Start by making sure Python is installed by verifying that the version is >3.10:
1014

1115
```bash
1216
python3 --version
@@ -34,10 +38,10 @@ From inside the `neural-graphics-model-gym-examples/` folder, run the setup scri
3438
./setup.sh
3539
```
3640

37-
This will:
38-
- create a Python virtual environment called `nb-env`
39-
- install the `ng-model-gym` package and required dependencies
40-
- download the datasets and weights needed to run the notebooks
41+
This will do the following:
42+
- Create a Python virtual environment called `nb-env`
43+
- Install the `ng-model-gym` package and required dependencies
44+
- Download the datasets and weights needed to run the notebooks
4145

4246
Activate the virtual environment:
4347

@@ -55,4 +59,5 @@ print("Torch version:", torch.__version__)
5559
print("Model Gym version:", ng_model_gym.__version__)
5660
```
5761

58-
You’re now ready to start walking through the training and evaluation steps.
62+
You’ve completed your environment setup - great work! You’re now ready to start walking through the training and evaluation steps.
63+

content/learning-paths/mobile-graphics-and-gaming/model-training-gym/3-model-training.md

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,19 +5,18 @@ weight: 4
55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
8+
## About NSS
89

910
In this section, you'll get hands-on with how you can use the model gym to fine-tune the NSS use-case.
1011

11-
## About NSS
12-
1312
Arm Neural Super Sampling (NSS) is an upscaling technique designed to solve a growing challenge in real-time graphics: delivering high visual quality without compromising performance or battery life. Instead of rendering every pixel at full resolution, NSS uses a neural network to intelligently upscale frames, freeing up GPU resources and enabling smoother, more immersive experiences on mobile devices.
1413

15-
The NSS model is available in two formats:
14+
The NSS model is available in two formats, as shown in the table below:
1615

1716
| Model format | File extension | Used for |
1817
|--------------|----------------|--------------------------------------------------------------------------|
19-
| PyTorch | .pt | training, fine-tuning, or evaluation in or scripts using the Model Gym |
20-
| VGF | .vgf | for deployment using ML Extensions for Vulkan on Arm-based hardware or emulation layers |
18+
| PyTorch | `.pt` | training, fine-tuning, or evaluation in or scripts using the Model Gym |
19+
| VGF | `.vgf` | for deployment using ML Extensions for Vulkan on Arm-based hardware or emulation layers |
2120

2221
Both formats are available in the [NSS repository on Hugging Face](https://huggingface.co/Arm/neural-super-sampling). You'll also be able to explore config files, model metadata, usage details and detailed documentation on the use-case.
2322

@@ -62,6 +61,8 @@ neural-graphics-model-gym-examples/tutorials/nss/model_evaluation_example.ipynb
6261

6362
At the end you should see a visual comparison of the NSS upscaling and the ground truth image.
6463

65-
Proceed to the final section to view the model structure and explore further resources.
64+
65+
You’ve completed the training and evaluation steps. Proceed to the final section to view the model structure and explore further resources.
66+
6667

6768

content/learning-paths/mobile-graphics-and-gaming/model-training-gym/4-model-explorer.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,27 +12,29 @@ Model Explorer is a visualization tool for inspecting neural network structures
1212

1313
This lets you inspect model architecture, tensor shapes, and graph connectivity before deployment. This can be a powerful way to debug and understand your exported neural graphics models.
1414

15-
## Setting up the VGF adapter
15+
## Set up the VGF adapter
1616

1717
The VGF adapter extends Model Explorer to support `.vgf` files exported from the Model Gym toolchain.
1818

19-
### Install the VGF adapter with pip
19+
## Install the VGF adapter with pip
20+
21+
Run:
2022

2123
```bash
2224
pip install vgf-adapter-model-explorer
2325
```
2426

25-
The source code is available on [GitHub](https://github.com/arm/vgf-adapter-model-explorer).
27+
The VGF adapter model explorer source code is available on [GitHub](https://github.com/arm/vgf-adapter-model-explorer).
2628

27-
### Install Model Explorer
29+
## Install Model Explorer
2830

2931
The next step is to make sure the Model Explorer itself is installed. Use pip to set it up:
3032

3133
```bash
3234
pip install torch ai-edge-model-explorer
3335
```
3436

35-
### Launch the viewer
37+
## Launch the viewer
3638

3739
Once installed, launch the explorer with the VGF adapter:
3840

@@ -44,6 +46,4 @@ Use the file browser to open the `.vgf` model exported earlier in your training
4446

4547
## Wrapping up
4648

47-
Through this Learning Path, you’ve learned what neural graphics is and why it matters for game performance. You’ve stepped through the process of training and evaluating an NSS model using PyTorch and the Model Gym, and seen how to export that model into VGF (.vgf) for real-time deployment. You’ve also explored how to visualize and inspect the model’s structure using Model Explorer.
48-
49-
As a next step, you can head over to the [Model Training Gym repository](https://github.com/arm/neural-graphics-model-gym/tree/main) documentation to explore integration into your own game development workflow. You’ll find resources on fine-tuning, deeper details about the training and export process, and everything you need to adapt to your own content and workflows.
49+
Through this Learning Path, you’ve learned what neural graphics is and why it matters for game performance. You’ve stepped through the process of training and evaluating an NSS model using PyTorch and the Model Gym, and seen how to export that model into VGF (.vgf) for real-time deployment. You’ve also explored how to visualize and inspect the model’s structure using Model Explorer. You can now explore the Model Training Gym repository for deeper integration and to keep building your skills.

content/learning-paths/mobile-graphics-and-gaming/model-training-gym/_index.md

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,6 @@
11
---
2-
title: Fine-Tuning Neural Graphics Models with Model Gym
3-
4-
draft: true
5-
cascade:
6-
draft: true
7-
2+
title: Fine-tuning neural graphics models with Model Gym
3+
84
minutes_to_complete: 45
95

106
who_is_this_for: This is an advanced topic for developers exploring neural graphics and interested in training and deploying upscaling models like Neural Super Sampling (NSS) using PyTorch and Arm’s hardware-aware backend.
@@ -50,10 +46,15 @@ further_reading:
5046
title: NSS on HuggingFace
5147
link: https://huggingface.co/Arm/neural-super-sampling
5248
type: website
49+
- resource:
50+
title: Vulkan ML Sample Learning Path
51+
link: /learning-paths/mobile-graphics-and-gaming/vulkan-ml-sample/
52+
type: learningpath
5353

5454

5555
### FIXED, DO NOT MODIFY
56-
weight: 1
57-
layout: "learningpathall"
58-
learning_path_main_page: "yes"
56+
# ================================================================================
57+
weight: 1 # _index.md always has weight of 1 to order correctly
58+
layout: "learningpathall" # All files under learning paths have this same wrapper
59+
learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
5960
---

0 commit comments

Comments
 (0)