Skip to content

Commit 548e647

Browse files
committed
feat: add example scripts and update documentation for CLI usage; introduce example1_cli.py and example_server.py, along with example.sh for environment setup and command execution
1 parent a6eccc2 commit 548e647

File tree

13 files changed

+76
-68
lines changed

13 files changed

+76
-68
lines changed

docs/readme_en.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
<!-- v0.1.2 -->
2+
13
<div align="center">
24
<img src="./fig/logo.png" width="50%" alt="FlowLine" />
35

@@ -51,7 +53,7 @@ The system uses a list file (`.xlsx`、 `.csv` or `.json` format) to define task
5153
<details>
5254
<summary>Example and Explanation</summary>
5355

54-
Example files: [`test/todo.xlsx`](./test/todo.xlsx), [`test/todo.csv`](./test/todo.csv),[`test/todo.json`](./test/todo.json), which can be constructed using the example program [`test/task_builder.py`](./test/task_builder.py).
56+
Example files: [`test/example1_todo.xlsx`](./test/example1_todo.xlsx), [`test/example1_todo.csv`](./test/example1_todo.csv),[`test/example1_todo.json`](./test/example1_todo.json), which can be constructed using the example program [`test/task_builder.py`](./test/task_builder.py).
5557

5658
| *name* | lr | batch_size | *run_num* | *need_run_num* | *cmd* |
5759
| --------- | ----- | ---------- | --------- | -------------- | ----------- |
@@ -88,7 +90,7 @@ if __name__ == "__main__":
8890
args = " ".join([f"--{k} {v}" for k, v in param_dict.items()])
8991
return cmd + args
9092

91-
run_cli(func, "test/todo.xlsx")
93+
run_cli(func, "test/example1_todo.xlsx")
9294
```
9395

9496
* `param_dict`: Dictionary built from current Excel row (keys=column names, values=cell content)
@@ -182,7 +184,7 @@ Besides CLI, you can use the Web GUI for **real-time monitoring and dynamic inte
182184
#### 1. Start Backend API Service
183185
Run the Flask backend:
184186
```bash
185-
python main_server.py
187+
python test/example_server.py
186188
```
187189

188190
#### 2. Start Frontend Service

example.sh

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# 先进入虚拟环境
2+
conda activate <你的虚拟环境名称>
3+
export PYTHONPATH=/home/me/project/flowline/ # 【可能需要修改】
4+
# 因为example放在了test目录下,所以需要设置PYTHONPATH
5+
# 正常情况下,cli/server在项目根目录下,所以不需要设置PYTHONPATH
6+
7+
# 命令行调用示例
8+
9+
python test/example1_cli.py
10+
python test/example2_cli.py
11+
12+
# 集成式服务器
13+
python test/example_server.py # 启动服务器
14+
cd web && python -m http.server 8000 # 在【新的终端】,启动前端静态文件服务器

readme.md

Lines changed: 14 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
<!-- # FlowLine -->
1+
<!-- v0.1.3 -->
22

33
<div align="center">
44
<img src="./docs/fig/logo.png" width="50%" alt="FlowLine" />
@@ -29,10 +29,12 @@ FlowLine 是一个用于 **GPU资源管理** 和 **并发指令流调度** 的
2929

3030
## Updates
3131

32-
* 2025.09.09: 新增用户自定义 GPU 优先级排序功能,通过 `user_cmp` 参数传入自定义排序函数,详见 [main_cli.py](./main_cli.py) 示例。
32+
* 2025.09.09: 新增用户自定义 GPU 优先级排序功能,通过 `user_cmp` 参数传入自定义排序函数,详见 [example1_cli.py](./test/example1_cli.py) 示例。
3333

3434
## 🚀 快速使用指南
3535

36+
> 更精炼的, [example.sh](./example.sh) 给出了所有示例。
37+
3638
### 🖥️ 使用命令行接口(CLI 模式)
3739

3840
#### 1. 安装方式
@@ -51,7 +53,7 @@ pip install fline
5153
pip install -e <flowline库路径>
5254
```
5355

54-
> 注:请确保你已安装 `pandas``psutil``openpyxl``requirements.txt`内的基本依赖。
56+
> 注:请确保你已安装 `pandas``psutil``openpyxl``requirements.txt`内的基本依赖。如果只用 CLI 方式可以不用安装 flash 相关包。
5557
5658
#### 2. 编写任务控制表
5759

@@ -60,7 +62,7 @@ pip install -e <flowline库路径>
6062
<details>
6163
<summary>示例和说明</summary>
6264

63-
示例文件:[`test/todo.xlsx`](./test/todo.xlsx)[`test/todo.csv`](./test/todo.csv)[`test/todo.json`](./test/todo.json), 可以通过运行[`test/task_builder.py`](./test/task_builder.py)示例构造程序构造。
65+
示例文件:[`test/example1_todo.xlsx`](./test/example1_todo.xlsx)[`test/example1_todo.csv`](./test/example1_todo.csv)[`test/example1_todo.json`](./test/example1_todo.json), 可以通过运行[`test/task_builder.py`](./test/task_builder.py)示例构造程序构造。
6466

6567
| *name* | lr | batch\_size |*run\_num*|*need\_run\_num*| *cmd* |
6668
| --------- | ----- | ----------- | -------- | -------------- | ----------- |
@@ -88,26 +90,18 @@ pip install -e <flowline库路径>
8890
<details>
8991
<summary>示例和说明</summary>
9092

91-
详见 [main_cli.py](./main_cli.py) 示例。主要部分如下:
93+
详见 [example1_cli.py](./test/example1_cli.py) 示例。主要部分如下:
9294

9395
```python
9496
def func(dict, gpu_id, sorted_gpu_ids):
95-
print(sorted_gpu_ids)
9697
return "CUDA_VISIBLE_DEVICES="+str(gpu_id)+" python -u test/test.py "+ " ".join([f"--{k} {v}" for k, v in dict.items()])
9798

98-
def cmp(info1, info2):
99-
if info1.free_memory > info2.free_memory:
100-
return -1
101-
elif info1.free_memory < info2.free_memory:
102-
return 1
103-
else:
104-
return 0
105-
10699
if __name__ == "__main__":
107-
# run_cli(func, "test/todo.csv")
108-
run_cli(func, "test/todo.csv", user_cmp=cmp) # user_cmp可选
100+
run_cli(func, "test/example1_todo.xlsx")
109101
```
110102

103+
对于直接执行完整命令的需求,见 [example2_cli.py](./test/example2_cli.py)
104+
111105
* `dict` 是由 Excel 中当前任务行构造出的字典,键为列名,值为单元格内容;
112106
* `gpu_id` 是系统动态分配的 GPU 编号,保证任务不冲突,其此刻一定满足显存空间最小限制;
113107
* `sorted_gpu_ids`(可选)是经过优先级排序后的可用 GP U序列,为可能的多 GPU 任务适配;
@@ -161,10 +155,10 @@ log/
161155

162156
#### 4. 运行程序后输入`run`开始运行任务流
163157

164-
运行 `main_cli.py` 启动程序:
158+
运行 `example1_cli.py` 启动程序:
165159

166160
```bash
167-
python main_cli.py
161+
python test/example1_cli.py
168162
```
169163

170164
<details>
@@ -227,10 +221,10 @@ python main_cli.py
227221

228222
### 1. 启动后端 API 服务
229223

230-
后端为 Flask 应用,运行 `main_server.py` 启动服务:
224+
后端为 Flask 应用,运行 `example_server.py` 启动服务:
231225

232226
```bash
233-
python main_server.py
227+
python test/example_server.py
234228
```
235229

236230
### 2. 启动前端界面服务

setup.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
from setuptools import setup, find_packages
22
import os
33

4-
base_version = "0.1.1"
4+
base_version = "0.1.2"
55
build_timestamp = os.getenv('BUILD_TIMESTAMP', None)
66

77
if build_timestamp:

main_cli.py renamed to test/example1_cli.py

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
# 命令行调用示例
2-
32
from flowline import run_cli
43

54
def func(dict, gpu_id, sorted_gpu_ids):
5+
# ! 这里需要修改为你的命令
6+
# 示例: 这里通过字典 dict 读出参数,所以需要使用 " ".join([f"--{k} {v}" for k, v in dict.items()]) 将参数拼接起来
7+
# 对于直接执行完整命令的需求,见 example2_cli.py
68
return "CUDA_VISIBLE_DEVICES="+str(gpu_id)+" python -u test/test.py "+ " ".join([f"--{k} {v}" for k, v in dict.items()])
79

810
def cmp(info1, info2):
@@ -14,7 +16,8 @@ def cmp(info1, info2):
1416
return 0
1517

1618
if __name__ == "__main__":
17-
# run_cli(func, "test/todo.csv")
18-
run_cli(func, "test/todo.csv", user_cmp=cmp) # user_cmp可选
19+
# ! 这里需要修改为你的任务配置文件路径
20+
run_cli(func, "test/example1_todo.csv", user_cmp=cmp)
21+
# user_cmp 可选 e.g. run_cli(func, "test/example1_todo.xlsx")
1922

2023
# 如果出现异常进程,可以使用 pkill -9 python 杀死所有进程
File renamed without changes.
File renamed without changes.
Binary file not shown.

test/example2_cli.py

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
from flowline.api import run_cli
2+
3+
def func(dict, gpu_id, sorted_gpu_ids):
4+
# ! 这里需要修改为你的命令
5+
# 示例: 这里是直接读出命令 mycmd 而不是使用参数,所以直接返回 dict["mycmd"]
6+
return "CUDA_VISIBLE_DEVICES="+str(gpu_id)+" "+dict["mycmd"]+" --discribe "+'"'+dict["description"]+'"'
7+
8+
if __name__ == "__main__":
9+
run_cli(func, "trash/todo.csv")
10+
11+
# 如果出现异常进程,可以使用 pkill -9 python 杀死所有进程

test/example2_todo.csv

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
description,mycmd,run_num,need_run_num,name,cmd
2+
Resnet50+自蒸馏 weather训练,python train.py train multi_resnet50_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.02 --use-self False,0,1,Task:0,No command
3+
Resnet50+自蒸馏 weather训练,python train.py train multi_resnet50_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.02,0,1,Task:1,No command
4+
Resnet50+频域Attention weather训练,python train.py train Spec_Resnet50_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --use-self False --lr 0.02,0,1,Task:2,No command
5+
Ours-small weather训练 (Resnet50+频域Attention+自蒸馏),python train.py train Spec_Resnet50_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.001 --alpha 0.1 --temperature 1,0,1,Task:3,No command
6+
Resnet101+自蒸馏 weather训练,python train.py train multi_resnet101_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.02 --use-self False,0,1,Task:4,No command
7+
Resnet101+自蒸馏 weather训练,python train.py train multi_resnet101_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.02,0,1,Task:5,No command
8+
Resnet101+频域Attention weather训练,python train.py train Spec_Resnet101_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --use-self False --lr 0.001,0,1,Task:6,No command
9+
Ours-base weather训练,python train.py train Spec_Resnet101_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.005,0,1,Task:7,No command
10+
Resnet152+自蒸馏 weather训练,python train.py train multi_resnet152_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.02 --use-self False,0,1,Task:8,No command
11+
Resnet152+自蒸馏 weather训练,python train.py train multi_resnet152_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.02,0,1,Task:9,No command
12+
Resnet152+频域Attention weather训练,python train.py train Spec_Resnet152_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.001 --use-self False,0,1,Task:10,No command
13+
Ours-large weather训练,python train.py train Spec_Resnet152_kd --data-dir /home/me/datasets/weather --dataset weather --batch-size 4 --print-freq 100 --lr 0.001,0,1,Task:11,No command

0 commit comments

Comments
 (0)