You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+50-23Lines changed: 50 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# TaskVine Report Tool
2
2
3
-
An interactive visualization tool for [TaskVine](https://cctools.readthedocs.io/en/stable/taskvine/), a task scheduler for large workflows to run efficiently on HPC clusters. This tool helps you analyze task execution patterns, file transfers, resource utilization, and other key metrics.
3
+
An interactive visualization tool for [TaskVine](https://github.com/cooperative-computing-lab/cctools), a task scheduler for large workflows to run efficiently on HPC clusters. This tool helps you analyze task execution patterns, file transfers, resource utilization, storage consumption, and other key metrics.
The easiest way to use this tool is to configure TaskVine to generate logs directly in the correct location. When creating your TaskVine manager, set these parameters:
19
+
After running your TaskVine workflow, you'll find the logs in a directory named with a timestamp (e.g., `2025-05-20T110437`) or your specified workflow name. The default structure looks like this:
20
+
21
+
```
22
+
workflow_name/
23
+
└── vine-logs/
24
+
├── debug
25
+
├── performance
26
+
├── taskgraph
27
+
├── transactions
28
+
└── workflow.json
29
+
```
30
+
31
+
To use these logs with the visualization tool:
32
+
33
+
1. Copy the entire workflow directory to the tool's `logs` directory:
34
+
```bash
35
+
cp -r /path/to/workflow_name logs/
36
+
```
37
+
38
+
2. Generate the visualization data:
39
+
```bash
40
+
python3 generate_data.py logs/your_workflow_name
41
+
```
42
+
43
+
3. Start the visualization server:
44
+
```bash
45
+
python3 app.py
46
+
```
47
+
48
+
4. View the report in your browser at `http://localhost:9122`
49
+
50
+
Note: In the web interface, you'll only see log collections that have been successfully processed by `generate_data.py`. You can process multiple log collections at once by providing multiple paths:
Instead of manually copying logs, you can configure TaskVine to generate logs directly in the correct location. When creating your TaskVine manager, set these parameters:
20
59
21
60
```python
22
61
manager = vine.Manager(
23
62
9123,
24
63
run_info_path="~/taskvine-report-tool", # Path to this tool's directory
25
-
run_info_template="experiment1"# Name for this run's logs
64
+
run_info_template="your_workflow_name"# Name for this run's logs
26
65
)
27
66
```
28
67
29
68
This will automatically create the correct directory structure:
30
69
```
31
70
~/taskvine-report-tool/
32
71
└── logs/
33
-
└── experiment1/
72
+
└── your_workflow_name/
34
73
└── vine-logs/
35
74
├── debug
36
75
└── transactions
37
76
```
38
77
39
78
After your workflow completes, simply:
40
79
1. Navigate to the tool directory: `cd ~/taskvine-report-tool`
41
-
2. Generate the visualization data: `python3 generate_data.py logs/experiment1`
80
+
2. Generate the visualization data: `python3 generate_data.py logs/your_workflow_name`
42
81
3. Refresh your browser to see the new log collection
43
82
44
-
### 2. Prepare Log Files
45
-
46
-
All log files should be placed in the `logs` directory. Currently, the tool only parses the `debug` and `transactions` logs, so the other log files are optional. The directory structure should be:
47
-
48
-
```
49
-
logs/
50
-
└── your_log_folder_name/
51
-
└── vine-logs/
52
-
├── debug (required)
53
-
├── performance (optional)
54
-
├── taskgraph (optional)
55
-
└── transactions (required)
56
-
```
83
+
### 3. Multiple Log Collections
57
84
58
-
For example, if you have multiple log collections, your directory structure might look like this:
85
+
You can have multiple log collections in your `logs`directory. For example:
59
86
60
87
```
61
88
logs/
@@ -75,7 +102,7 @@ logs/
75
102
└── transactions
76
103
```
77
104
78
-
### 3. Generate Visualization Data
105
+
### 4. Generate Visualization Data
79
106
80
107
For each log collection, generate the visualization data by running:
81
108
@@ -103,7 +130,7 @@ logs/
103
130
└── subgraphs.pkl # Task dependency graphs
104
131
```
105
132
106
-
### 4. Start Visualization Server
133
+
### 5. Start Visualization Server
107
134
108
135
After generating the data, start the web server:
109
136
@@ -117,7 +144,7 @@ By default, the server runs on port 9122. You can specify a different port using
117
144
python3 app.py --port 8080
118
145
```
119
146
120
-
### 5. View Visualization Report
147
+
### 6. View Visualization Report
121
148
122
149
Once the server is running, access the visualization in your browser at:
0 commit comments