Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@
<a href="https://github.com/user-attachments/assets/125e3494-25c9-4494-9acd-8ad65ca85d03"><img src="https://img.shields.io/badge/微信-green?logo=wechat&amp"></a>
</div>

**GraphNet** is a large-scale dataset of deep learning **computation graphs**, built as a standard benchmark for **tensor compiler** optimization. It provides over 2.7K computation graphs extracted from state-of-the-art deep learning models spanning diverse tasks and ML frameworks. With standardized formats and rich metadata, GraphNet enables fair comparison and reproducible evaluation of the general optimization capabilities of tensor compilers, thereby supporting advanced research such as AI for System on compilers.
**GraphNet** is a large-scale dataset of deep learning **computation graphs**, built as a standard benchmark for **tensor compiler** optimization. It provides over 2.7K computation graphs extracted from state-of-the-art deep learning models spanning diverse tasks and ML frameworks. With standardized formats and rich metadata, GraphNet enables fair comparison and reproducible evaluation of the general optimization capabilities of tensor compilers, thereby supporting advanced research such as AI for System on Compilers.

## 📣 News
- [2025-11-19] ✨ Keynote Speech at GTOC Forum 2025: [GraphNet Empowering the AI Software Stack]( https://b23.tv/PFzSKK1)
- [2025-10-14] ✨ Our technical report is out: a detailed study of dataset construction and compiler benchmarking, introducing the novel performance metrics Speedup Score S(t) and Error-aware Speedup Score ES(t). [📘 GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research](https://arxiv.org/abs/2510.24035)
- [2025-8-20] 🚀 The second round of [open contribution tasks](https://github.com/PaddlePaddle/Paddle/issues/74773) was released. (completed ✅)
- [2025-7-30] 🚀 The first round of [open contribution tasks](https://github.com/PaddlePaddle/GraphNet/issues/44) was released. (completed ✅)
Expand Down Expand Up @@ -84,7 +85,7 @@ python -m graph_net.plot_violin \
--output-dir $GRAPH_NET_BENCHMARK_PATH
```

The scripts are designed to process a file structure as `/benchmark_path/category_name/`, and items on x-axis are identified by name of the sub-directories. After executing, several summary plots of result in categories (model tasks, libraries...) will be exported to `$GRAPH_NET_BENCHMARK_PATH`.
The scripts are designed to process a file structure as `/benchmark_path/category_name/`, and items on the x-axis are identified by the name of the sub-directories. After executing, several summary plots of results in categories (model tasks, libraries...) will be exported to `$GRAPH_NET_BENCHMARK_PATH`.

### Hardware Regression Testing
We also provide a two-step workflow that validates compiler correctness and performance against a "golden" reference, which is crucial for hardware-specific testing and regression tracking. Details can be found in this [guide](./docs/hardware_test.md).
Expand All @@ -105,7 +106,7 @@ Check out the [Construction Guide](./docs/README_contribute.md) for details on t

## GraphNet Community

You can join our community via following group chats. Welcome to ask any questions about using and building GraphNet.
You can join our community by following the group chats. Welcome to ask any questions about using and building GraphNet.

<div align="center">
<table>
Expand Down
138 changes: 138 additions & 0 deletions README_cn.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@

<h1 align="center">GraphNet:面向张量编译器研究的大规模计算图数据集</h1>

<div align="center">

![](https://img.shields.io/github/issues/PaddlePaddle/GraphNet?label=open%20issues)
[![arXiv](https://img.shields.io/badge/arXiv-2510.24035-b31b1b.svg)](https://arxiv.org/abs/2510.24035)
<a href="https://github.com/user-attachments/assets/125e3494-25c9-4494-9acd-8ad65ca85d03"><img src="https://img.shields.io/badge/微信-green?logo=wechat&amp"></a>
</div>

**GraphNet** 是一个大规模深度学习**计算图数据集**,旨在为**张量编译器**优化提供一个标准的基准测试平台。它包含了从覆盖多种任务和机器学习框架的先进深度学习模型中提取的超过 2700个 计算图。凭借其标准化的格式和丰富的元数据,GraphNet 能够对张量编译器的通用优化能力进行公平比较和可复现的评估,从而支持诸如面向编译器的“AI for System”等前沿研究。

## 📣 最新动态
- [2025-11-19] ✨ 在 GTOC Forum 2025 上的主题演讲:[GraphNet 助力 AI 软件栈催熟](https://b23.tv/PFzSKK1)
- [2025-10-14] ✨ 我们的技术报告已发布:这是一份关于数据集构建和编译器基准测试的详细研究,并引入了新颖的性能指标——加速分数 S(t) 和感知错误的加速分数 ES(t)。[📘 GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research](https://arxiv.org/abs/2510.24035)
- [2025-8-20] 🚀 第二轮 [开源贡献任务](https://github.com/PaddlePaddle/Paddle/issues/74773) 已发布。(已完成 ✅)
- [2025-7-30] 🚀 第一轮 [开源贡献任务](https://github.com/PaddlePaddle/GraphNet/issues/44) 已发布。(已完成 ✅)
## 📊 基准测试结果
我们在 GraphNet 的 NLP 和 CV 子集上评估了两个代表性的张量编译器后端:CINN (PaddlePaddle) 和 TorchInductor (PyTorch)。评估采用了[技术报告](https://arxiv.org/abs/2510.24035)中提出的两个量化指标:
- **加速分数** S(t) — 评估编译器在不同数值容忍度下的性能。
<div align="center">
<img src="/pics/St-result.jpg" alt="Speedup Score S_t Results" width="80%">
</div>

- **感知错误的加速分数** ES(t) — 进一步考量运行时和编译错误。
<div align="center">
<img src="/pics/ESt-result.jpg" alt="Error-aware Speedup Score ES_t Results" width="80%">

</div>

## ⚡ 快速开始
本节面向编译器用户/开发者展示如何评估张量编译器并复现基准测试结果,以及面向 GraphNet 贡献者展示如何贡献新的计算图。

### ⚖️ 编译器评估

**步骤 1:基准测试**

使用 `graph_net.torch.test_compiler` 对 GraphNet 样本进行基准测试,可指定批次和日志配置:

```bash
# 设置你的基准测试目录
export GRAPH_NET_BENCHMARK_PATH=/home/yourname/graphnet_benchmark/

# 运行基准测试
python -m graph_net.torch.test_compiler \
--model-path $GRAPH_NET_EXTRACT_WORKSPACE/model_name/ \
--compiler /custom/or/builtin/compiler/ \
--device /device/to/execute/ \
--warmup /times/to/warmup/ \
--trials /times/to/test/ \
> $GRAPH_NET_BENCHMARK_PATH/log.log 2>&1

# 注意:如果省略 --compiler 参数,默认使用 PyTorch 的内置编译器。
```

执行后,`graph_net.torch.test_compiler` 将:
1. 以即时执行模式运行原始模型,记录基线性能。
2. 使用指定的后端(例如 CINN, TVM, Inductor, TensorRT, XLA, BladeDISC)编译模型。
3. 执行编译后的模型,收集其运行时间和输出。
4. 若无执行失败,则将编译结果与基线对比,计算加速比。

**步骤 2:分析**

使用 `graph_net.plot_St`、`graph_net.plot_ESt` 和 `graph_net.plot_violin` 这三个脚本,根据基准测试日志中的加速比、正确性和运行时信息,生成 St 图、ESt 图和 [小提琴图](https://en.m.wikipedia.org/wiki/Violin_plot)。

```bash
python -m graph_net.plot_St \
--benchmark-path $GRAPH_NET_BENCHMARK_PATH/log.log \
--output-dir $GRAPH_NET_BENCHMARK_PATH \
--negative-speedup-penalty penalty/power/for/negative/speedup \
--fpdb base/penalty/for/severe/errors

python -m graph_net.plot_ESt \
--benchmark-path $GRAPH_NET_BENCHMARK_PATH/log.log \
--output-dir $GRAPH_NET_BENCHMARK_PATH \
--negative-speedup-penalty penalty/power/for/negative/speedup \
--fpdb base/penalty/for/severe/errors

# 注意:如果省略 --negative-speedup-penalty 参数,默认使用 p=0。
# 如果省略 --fpdb 参数,默认使用 b=0.1。

python -m graph_net.plot_violin \
--benchmark-path $GRAPH_NET_BENCHMARK_PATH/JSON_results/ \
--output-dir $GRAPH_NET_BENCHMARK_PATH
```

这些脚本设计用于处理 `/benchmark_path/category_name/` 这样的文件结构,x 轴上的项目由子目录名称标识。执行后,按类别(模型任务、库等)划分的结果汇总图表将被导出到 `$GRAPH_NET_BENCHMARK_PATH`。

### 硬件回归测试
我们还提供了一个两步工作流,用于根据“黄金标准”参考验证编译器的正确性和性能,这对于硬件专用测试和回归跟踪至关重要。详情可参阅 [指南](./docs/hardware_test_cn.md)。

### 🧱 构建与贡献指南
想了解 GraphNet 如何构建或贡献新样本?查看 [构建指南](./docs/README_contribute_cn.md) 以获取有关提取和验证工作流的详细信息。

## 🚀 未来路线图

1. 将 GraphNet 扩展至 10,000+ 计算图。
2. 为 GraphNet 样本添加更精细的子类别注释。
3. 从多 GPU 场景中提取样本,以支持大规模分布式计算的基准测试和优化。
4. 支持将完整计算图拆分为可独立优化的子图和算子序列。

**愿景**: GraphNet 旨在通过对张量编译器优化进行**大规模、系统性**的评估,并**为模型学习和迁移优化策略提供数据集**,从而为“面向编译器的 AI (AI for Compiler)”奠定基础。

## GraphNet 社区

您可以通过扫描下方群聊二维码加入我们的社区。欢迎提出任何关于使用和构建 GraphNet 的问题。

<div align="center">
<table>
<tr>
<td align="center">
<img width="200" src="https://github.com/user-attachments/assets/125e3494-25c9-4494-9acd-8ad65ca85d03" />
</td>
<td align="center">
<img width="150" src="https://cdn.prod.website-files.com/6257adef93867e50d84d30e2/67d00cf7266d2c75571aebde_Example.svg" />
<p><a href="https://discord.gg/vyeAydwh">Channel</a> is also available.</p>
</td>
</tr>
</table>
</div>

## 许可证与致谢

GraphNet 基于 [MIT 许可证](./LICENSE) 开源发布。

如果您觉得本项目对您的研究或工作有帮助,请引用:

```bibtex
@misc{li2025graphnetlargescalecomputationalgraph,
title={GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research},
author={Xinqi Li and Yiqun Liu and Shan Jiang and Enrong Zheng and Huaijin Zheng and Wenhao Dai and Haodong Deng and Dianhai Yu and Yanjun Ma},
year={2025},
eprint={2510.24035},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2510.24035},
}
```
8 changes: 4 additions & 4 deletions docs/README_contribute.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ export GRAPH_NET_EXTRACT_WORKSPACE=/home/yourname/graphnet_workspace/
# Extract the ResNet‑18 computation graph
python graph_net/test/vision_model_test.py

# Validate the extracted graph (e.g. /home/yourname/graphnet_workspace/resnet18/)
# Validate the extracted graph (e.g.,/home/yourname/graphnet_workspace/resnet18/)
python -m graph_net.torch.validate \
--model-path $GRAPH_NET_EXTRACT_WORKSPACE/resnet18/
```
Expand All @@ -45,7 +45,7 @@ Wrap the model with the extractor — that’s all you need:
```bash
import graph_net

# Instantiate the model (e.g. a torchvision model)
# Instantiate the model (e.g., a torchvision model)
model = ...

# Extract your own model
Expand All @@ -58,14 +58,14 @@ For more details, see docstring of `graph_net.torch.extract` defined in `graph_n

**Step 2: graph_net.torch.validate**

To verify that the extracted model meets requirements, we use `graph_net.torch.validate` in CI tool and also ask contributors to self-check in advance:
To verify that the extracted model meets requirements, we use `graph_net.torch.validate` in the CI tool and also ask contributors to self-check in advance:

```bash
python -m graph_net.torch.validate \
--model-path $GRAPH_NET_EXTRACT_WORKSPACE/model_name
```

All the **construction constraints** will be examined automatically. After passing validation, a unique `graph_hash.txt` will be generated and later checked in CI procedure to avoid redundant.
All **construction constraints** will be examined automatically. After passing validation, a unique `graph_hash.txt` will be generated and later checked in the CI procedure to avoid redundancy.

## 📁 Repository Structure
This repository is organized as follows:
Expand Down
87 changes: 87 additions & 0 deletions docs/README_contribute_cn.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# 为 GraphNet 做贡献
为确保数据集整体质量、可复现性及跨编译器兼容性,我们定义了以下构建**约束条件**:

1. 可运行:计算图必须在命令式(即时)模式下可执行。
2. 可序列化:计算图及其对应的 Python 代码必须支持序列化与反序列化。
3. 可分解:完整计算图应能分解为两个不相交的子图。
4. 可静态分析:每个计算图内的算子名称必须可静态解析。
5. 自定义算子可访问:若使用了自定义算子,其实现代码必须完全可访问。

## 图提取与验证
GraphNet 提供了用于图提取和验证的自动化工具。

<div align="center">
<img src="/pics/graphnet_overview.jpg" alt="GraphNet Architecture Overview" width="65%">
</div>

**示例:提取并验证 ResNet‑18**
```bash
git clone https://github.com/PaddlePaddle/GraphNet.git
cd GraphNet

# 设置您的工作空间目录
export GRAPH_NET_EXTRACT_WORKSPACE=/home/yourname/graphnet_workspace/

# 提取 ResNet-18 计算图
python graph_net/test/vision_model_test.py

# 验证提取的图(例如 /home/yourname/graphnet_workspace/resnet18/)
python -m graph_net.torch.validate \
--model-path $GRAPH_NET_EXTRACT_WORKSPACE/resnet18/
```

**工作流程图示说明**

<div align="center">
<img src="/pics/dataset_composition.png" alt="GraphNet Extract Sample" width="65%">
</div>

> 注:**仅当**模块中使用了相应的自定义算子时,才需要提供 custom_op 的源代码,且**对其格式无特定要求**。

**步骤 1:`graph_net.torch.extract`**

使用提取器包装您的模型——这就是您需要做的全部:

```bash
import graph_net

# 实例化模型(例如一个 torchvision 模型)
model = ...

# 提取您自己的模型
model = graph_net.torch.extract(name="model_name", dynamic=True)(model)
```

运行后,提取的计算图将保存至:`$GRAPH_NET_EXTRACT_WORKSPACE/model_name/`。

更多详细信息,请参阅 `graph_net/torch/extractor.py` 中定义的 `graph_net.torch.extract` 的文档字符串。

**步骤 2:`graph_net.torch.validate`**

为验证提取的模型是否符合要求,我们在 CI 工具中使用 `graph_net.torch.validate`,同时也要求贡献者提前进行自检:

```bash
python -m graph_net.torch.validate \
--model-path $GRAPH_NET_EXTRACT_WORKSPACE/model_name
```

所有**构建约束条件**都将被自动检查。通过验证后,将生成一个唯一的 `graph_hash.txt` 文件,后续在 CI 流程中会检查此文件以避免重复。

## 📁 代码仓库结构
本仓库结构如下:

| 目录 | 描述 |
|------------|--------------|
| **graph_net/** | 图提取、验证与基准测试的核心模块 |
| **paddle_samples/** | 从 PaddlePaddle 提取的计算图样本 |
| **samples/** | 从 PyTorch 提取的计算图样本 |
| **docs/** | 技术文档与贡献者指南|

以下是 **graph_net/** 目录的结构:
```text
graph_net/
├─ config/ # 配置文件、参数
├─ paddle/ # PaddlePaddle 图提取与验证
├─ torch/ # PyTorch 图提取与验证
├─ test/ # 单元测试与示例脚本
└─ *.py # 基准测试与分析脚本
4 changes: 2 additions & 2 deletions docs/hardware_test.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@ python -m graph_net.paddle.test_reference_device \
# --compiler: Specifies the compiler backend.
```
### Step 2: Run Regression Test
After changing hardware, run the correctness test script. This script reads the reference data, re-runs the models using the exact same configuration, and compares the new results against the "golden" reference.
After changing hardware, run the correctness test script. This script reads the reference data, re-runs the models using the same configuration, and compares the new results against the "golden" reference.
```bash
python -m graph_net.paddle.test_device_correctness \
--reference-dir ./golden_reference \
--device cuda
```
This script will report any failures (e.g., compilation errors, output mismatches) and print a performance comparison (speedup/slowdown) against the reference log, allowing you to quickly identify regressions.
This script will report any failures (e.g., compilation errors, output mismatches) and print a performance comparison (speedup/slowdown) against the reference log, allowing you to identify regressions quickly.
21 changes: 21 additions & 0 deletions docs/hardware_test_cn.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
## 硬件回归测试
### 步骤 1:生成参考数据
首先,在可信的环境中(例如,特定的硬件/编译器版本),使用 `graph_net.paddle.test_reference_device` 来生成基线日志和输出文件。
```bash
python -m graph_net.paddle.test_reference_device \
--model-path /path/to/all_models/ \
--reference-dir ./gold_reference \
--compiler cinn \
--device cuda

# --reference-dir: (必选)用于保存输出的 .log(性能/配置)文件和 .pdout(输出张量)文件的目录。
# --compiler: 指定编译器后端。
```
### 步骤 2:运行回归测试
更换硬件后,运行正确性测试脚本。此脚本会读取参考数据,使用完全相同的配置重新运行模型,并将新结果与“黄金”参考标准进行比对。
```bash
python -m graph_net.paddle.test_device_correctness \
--reference-dir ./golden_reference \
--device cuda
```
此脚本将报告任何失败情况(例如,编译错误、输出不匹配),并打印与参考日志的性能对比(加速/减速),从而帮助您快速识别性能回归问题。