You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
**GraphNet** is a large-scale dataset of deep learning **computation graphs**, built as a standard benchmark for **tensor compiler** optimization. It provides 2.7K computation graphs extracted from state-of-the-art deep learning models spanning diverse tasks and ML frameworks. With standardized formats and rich metadata, GraphNet enables fair comparison, reproducible evaluation, and deeper research into the general optimization capabilities of tensor compilers.
4
+
**GraphNet** is a large-scale dataset of deep learning **computation graphs**, built as a standard benchmark for **tensor compiler** optimization. It provides 2.7K computation graphs extracted from state-of-the-art deep learning models spanning diverse tasks and ML frameworks. With standardized formats and rich metadata, GraphNet enables fair comparison and reproducible evaluation of the general optimization capabilities of tensor compilers, thereby supporting advanced research in AI for compilers (**AI4C**).
<imgsrc="/pics/Eval_result.png"alt="Violin plots of speedup distributions"width="65%">
8
9
</div>
9
10
10
-
With GraphNet, users can:
11
-
1.**Contribute new computation graphs** through the built-in automated extraction and validation pipeline.
12
-
2.**Evaluate tensor compilers** on existing graphs with the integrated compiler evaluation tool, supporting multiple compiler backends.
13
-
3.**Advance research** in tensor compiler optimization using the test data and statistics provided by GraphNet.
14
-
15
-
16
-
11
+
Compiler developers can use GraphNet samples to evaluate tensor compilers (e.g., CINN, TorchInductor, TVM) on target tasks. The figure above shows the speedup of two compilers (CINN and TorchInductor) across two tasks (CV and NLP).
17
12
18
-
**Vision**: We aim to achieve cross-hardware portability of compiler optimizations by allowing models to learn and transfer optimization strategies. It will significantly reduce the manual effort required to develop efficient operator implementations.
19
13
20
14
21
15
## Dataset Construction
22
16
23
17
To guarantee the dataset’s overall quality, reproducibility, and cross-compiler compatibility, we define the following construction **constraints**:
24
18
25
-
1.Dynamic graphs must execute correctly.
26
-
2.Graphs and their corresponding Python code must support serialization and deserialization.
19
+
1.Computation graphs must be executable in imperative (eager) mode.
20
+
2.Computation graphs and their corresponding Python code must support serialization and deserialization.
27
21
3. The full graph can be decomposed into two disjoint subgraphs.
28
22
4. Operator names within each computation graph must be statically parseable.
29
23
5. If custom operators are used, their implementation code must be fully accessible.
30
24
31
25
32
26
### Graph Extraction & Validation
33
-
For full implementation details, please refer to the [Co-Creation Tutorial](https://github.com/PaddlePaddle/GraphNet/blob/develop/CONTRIBUTE_TUTORIAL.md#co-creation-tutorial).
27
+
28
+
We provide automated extraction and validation tools for constructing this dataset.
The compiler evaluation process takes a GraphNet sample as input and involves:
79
-
1. Running the original model in eager mode to record a baseline.
80
-
2. Compiling the model with the specified backend (e.g., CINN, TorchInductor, TVM).
81
-
3. Executing the compiled model and collecting its runtime and outputs.
82
-
4. Analyzing performance by comparing the compiled results against the baseline.
83
-
84
-
### Evaluation Metrics
85
-
86
-
We define two key metrics here: **rectified speedup** and **GraphNet Score**. Rectified speedup measures runtime performance while incorporating compilation success, time cost, and correctness. GraphNet Score aggregates the rectified speedup of a compiler on specified tasks, providing a measure of its general optimization capability.
87
-
88
78
**Demo: How to benchmark your compiler on the model:**
89
79
90
-
1. Benchmark
80
+
**Step 1: Benchmark**
91
81
92
82
We use ```graph_net/benchmark_demo.sh``` to benchmark GraphNet computation graph samples:
# Note: if --compiler is omitted, PyTorch’s built-in compiler is used by default
108
98
```
109
99
110
-
2. Analysis
100
+
**Step 2: Analysis**
111
101
112
102
After processing, we provide ```graph_net/analysis.py``` to generate [violin plot](https://en.m.wikipedia.org/wiki/Violin_plot) based on the JSON results.
113
103
@@ -121,20 +111,15 @@ After executing, one summary plot of results on all compilers (as shown below in
121
111
122
112
The script is designed to process a file structure as ```/benchmark_path/compiler_name/category_name/``` (for example ```/benchmark_logs/paddle/nlp/```), and items on x-axis are identified by name of the folders. So you can modify ```read_all_speedups``` function to fit the benchmark settings on your demand.
123
113
124
-
### Evaluation Results Example
125
-
126
-
<divalign="center">
127
-
<imgsrc="/pics/Eval_result.png"alt="Violin plots of rectified speedup distributions"width="65%">
128
-
</div>
129
-
130
-
131
114
## Roadmap
132
115
133
116
1. Scale GraphNet to 10K+ graphs.
134
117
2. Further annotate GraphNet samples into more granular sub-categories
135
118
3. Extract samples from multi-GPU scenarios to support benchmarking and optimization for large-scale, distributed computing.
136
119
4. Enable splitting full graphs into independently optimized subgraphs and operator sequences.
137
120
121
+
**Vision**: GraphNet aims to lay the foundation for AI4C by enabling large-scale, systematic evaluation of tensor compiler optimizations.
0 commit comments