Vine: Update Taskgraph Format and Tool (#4243)

* Add description of taskgraph file format

* Modify taskgraph log to generate abstract format, rather than writing graphviz dot directly.

* Add vine_plot_taskgraph to consume new taskgraph format.

* Organize plotting section to present each tool consistently:
debug, performance, taskgraph, transactions.

* Debug log first.

* format

* lint

---------

Co-authored-by: Benjamin Tovar <[email protected]>

Author: dthain

doc/manuals/taskvine/index.md

@@ -2455,9 +2455,8 @@ cores, memory and disk have modifiers `~` and `>` as follows:
 
 ## Logging, Plotting, and Tuning
 
-A TaskVine manager produces several logs: `debug`, `taskgraph`, `performance`,
-and `transactions`. These logs are always enabled, and appear in the current
-working directory in the subdirectories:
+A TaskVine manager produces several logs: `performance`, `taskgraph`, `debug`, and `transactions`.
+These logs are always enabled, and appear in the current working directory in the subdirectories:
 
 ```sh
 vine-run-info/YYYY-mm-ddTHH:MM:SS/vine-logs
@@ -2487,24 +2486,24 @@ If set, the environment variable `VINE_RUNTIME_INFO_DIR` determines the logging
 directory. If `VINE_RUNTIME_INFO_DIR` is not an absolute path, then it is taken
 relative to the current logging prefix (i.e. `vine-run-info/` by default).
 
-
 ### Debug Log
 
 The debug log prints unstructured messages as the manager transfers files and
 tasks, workers connect and report resources, etc. This is specially useful to
-find failures, bugs, and other errors. It is located by default at:
+find failures, bugs, and other errors when your workflow isn't running as expected.
+It is located by default at:
 
 ```sh
 vine-run-info/%Y-%m-%dT%H:%M:%S/vine-logs/debug
 ```
 
-To enable debugging at the worker, set the `-d` option:
-
+To view the debug log of the manager:
 ```sh
-$ vine_worker -d all -o worker.debug -M myproject
+less vine-run-info/most-recent/vine-logs/debug
 ```
 
-Custom APPLICATION messages can be added to the log with the calls:
+When writing an application, you may find it helpful to add your
+own application-level messages to the log with code like this:
 
 === "Python"
     ```python
@@ -2515,29 +2514,42 @@ Custom APPLICATION messages can be added to the log with the calls:
     ```
     vine_log_debug_app("your custom log message")
     ```
+You may also find it helpful to view the debug log of a worker,
+when running it locally. To enable debugging at the worker, add
+the `-d all` option:
+
+```sh
+$ vine_worker -d all -o worker.debug -M myproject
+```
+
+And then view the debug output of the worker process:
+```sh
+less worker.debug
+```
 
 ### Performance Log
 
 The performance log contains a time series of the statistics collected by the manager,
 such as number of tasks waiting and completed, number of workers busy,
-total number of cores available, etc. The log is located by default at:
+total number of cores available, etc.
+It is located by default at:
 
 ```sh
 vine-run-info/%Y-%m-%dT%H:%M:%S/vine-logs/performance
 ```
 
-Install gnuplot to enable the use of `vine_graph_log`
-
-```sh
-conda install conda-forge::gnuplot
-```
+!!! note
+    To use `vine_graph_log`, you must first install the `gnuplot` package like this:
+    ```
+    conda install conda-forge::gnuplot
+    ```
 
-The script `vine_graph_log` is a wrapper for `gnuplot`, and with it you
-can plot some of the statistics, such as total time spent transferring tasks,
-number of tasks running, and workers connected.  For example, this command:
+`vine_graph_log` will plot a wide variety of statistics from the performance log,
+such as total time spent transferring tasks, number of tasks running, and workers connected.
+For example, this command:
 
 ```sh
-$ vine_graph_log -o myplots my.stats.log
+$ vine_graph_log -o myplots vine-run-info/most-recent/vine-logs/performance
 ```
 
 produces the following graphs:
@@ -2546,11 +2558,41 @@ produces the following graphs:
 
 - [Performance Log File Format Details](log-file-formats.md#performance-log-format)
 
+### Taskgraph Log
+
+The `taskgraph` log file captures the graph relationship between every task and file in a workflow as it runs.
+To generate a visual graph from the log file, run `vine_plot_taskgraph`:
+It is located by default at:
+
+```sh
+vine-run-info/%Y-%m-%dT%H:%M:%S/vine-logs/taskgraph
+```
+
+!!! note
+    To use `vine_plot_taskgraph`, you must first install the `graphviz` package like this:
+    ```
+    conda install conda-forge::graphviz
+    ```
+
+```sh
+vine_plot_taskgraph vine-run-info/most-recent/vine-logs/taskgraph --output taskgraph.pdf
+```
+
+Which produces a graph like the following:
+
+![Example Task Graph](images/plot-taskgraph.png)
+
+Note that very large task graphs may take a long time to render, and make be impractical to display
+if there are a large number of edges to be rendered.
+
+- [Taskgraph Log File Format Details](log-file-formats.md#taskgraph-log-format)
+
 ### Transactions Log
 
 The transactions log records the lifetime of tasks and workers. It is
 specially useful for tracking the resources requested, allocated, and used by
-specific tasks. It is located by default at:
+specific tasks.
+It is located by default at:
 
 ```sh
 vine-run-info/%Y-%m-%dT%H:%M:%S/vine-logs/transactions
@@ -2570,7 +2612,6 @@ to produce a visualization of how tasks are packed into workers like this:
 
 - [Transactions Log File Format Details](log-file-formats.md#transactions-log-format)
 
-
 Custom APPLICATION messages can be added to the log with the calls:
 
 === "Python"
@@ -2667,28 +2708,6 @@ For the *workers* file:
 | `disk` | Total disk space in MB available at the worker.|
 | `gpus` | Total number of gpus available at the worker.|
 
-### Task Graph Log
-
-The complete graph of tasks and files is recorded in `taskgraph`
-using the [Graphviz](https://graphviz.org) Dot file format.  With the `dot` tool installed, you
-can visualize the task graph as follows:
-
-```sh
-dot -Tpng vine-run-info/most-recent/vine-logs/taskgraph > taskgraph.png
-```
-
-This can produce results like this:
-
-![Example Task Graph](images/plot-taskgraph.png)
-
-Note that very large task graphs may be impractical to graph at this level of detail.
-
-!!! note
-    You may need to install Graphviz Dot separately like this:
-    ```
-    conda install -c conda-forge graphviz
-    ```
-
 ### TaskVine Report Tool
 
 The [TaskVine Report Tool](https://github.com/cooperative-computing-lab/taskvine-report-tool) is a separate repository maintained by our team that provides web-based visualization capabilities. It is highly optimized for parsing large log files and generating informative visualizations. Here are some screenshots of this tool:

doc/manuals/taskvine/log-file-formats.md

@@ -19,6 +19,35 @@ Here is an example of the first few rows and columns:
 ...
 ```
 
+## Taskgraph Log Format
+
+The `taskgraph` log contains a sequence of records describing the basic
+properties of each file and task, and the relationships between the two.
+It is suitable for determining the provenance of files and tasks, and
+can be used to display the visual structure of a workflow.
+
+Each line is ASCII text with words delimited by spaces, except where quoted.
+Lines beginning with `#` are comments.  A file record consists of the word
+`FILE` followed by the unique file id, the quoted source path of the file,
+and the size of the file, if known.
+A task record consists of the word `TASK` followed by the unique task ID,
+the quoted name of the task, the keyword `INPUTS` followed by a list of input file IDs,
+and the keyword `OUTPUTS` followed by a list of output file IDs.
+
+```text
+# taskvine taskgraph version 2
+# TASK taskid "program" INPUTS fileid1 fileid fileid3 ... OUTPUTS fileid4 fileid5 ...
+# FILE fileid "source" size
+FILE file-meta-9c24a99af02ce7f488fcc7461fba2423 "convert.sfx" 350768
+FILE url-rnd-qjmltcbhuctjrfl "A-Cat.jpg" 163047
+FILE temp-rnd-sfscsbmjstfkvob "temp" 0
+TASK T1 "convert.sfx" INPUTS file-meta-9c24a99af02ce7f488fcc7461fba2423 url-rnd-qjmltcbhuctjrfl OUTPUTS temp-rnd-sfscsbmjstfkvob 
+TASK T4 "convert.sfx" INPUTS file-meta-9c24a99af02ce7f488fcc7461fba2423 url-rnd-qjmltcbhuctjrfl OUTPUTS temp-rnd-wqtpljproftkoym 
+TASK T23 "convert.sfx" INPUTS file-meta-9c24a99af02ce7f488fcc7461fba2423 url-rnd-qjmltcbhuctjrfl OUTPUTS temp-rnd-ycbskigfhodnnuj 
+TASK T5 "convert.sfx" INPUTS file-meta-9c24a99af02ce7f488fcc7461fba2423 url-rnd-qjmltcbhuctjrfl OUTPUTS temp-rnd-evtwiteajlpgwjn 
+...
+```
+
 ## Transactions Log Format
 
 The first few lines of the log document the possible log records:

taskvine/src/manager/vine_taskgraph_log.c

@@ -3,17 +3,14 @@
 #include "vine_mount.h"
 #include "vine_task.h"
 
-#include "path.h"
-
 #include <inttypes.h>
 #include <stdio.h>
 
-static int show_names = 0;
-
 void vine_taskgraph_log_write_header(struct vine_manager *q)
 {
-	fprintf(q->graph_logfile, "digraph \"taskvine\" {\n");
-	fprintf(q->graph_logfile, "node [style=filled,font=Helvetica,fontsize=10];\n");
+	fprintf(q->graph_logfile, "# taskvine taskgraph version 2\n");
+	fprintf(q->graph_logfile, "# TASK taskid \"program\" INPUTS fileid1 fileid fileid3 ... OUTPUTS fileid4 fileid5 ...\n");
+	fprintf(q->graph_logfile, "# FILE fileid \"source\" size\n");
 }
 
 void vine_taskgraph_log_write_task(struct vine_manager *q, struct vine_task *t)
@@ -28,21 +25,25 @@ void vine_taskgraph_log_write_task(struct vine_manager *q, struct vine_task *t)
 	if (p)
 		*p = 0;
 
-	fprintf(q->graph_logfile, "\"task-%d\" [color=green,label=\"%s\"];\n", id, show_names ? path_basename(name) : "");
+	fprintf(q->graph_logfile, "TASK T%d \"%s\" INPUTS ", id, name);
 
 	free(name);
 
 	struct vine_mount *m;
 
 	LIST_ITERATE(t->input_mounts, m)
 	{
-		fprintf(q->graph_logfile, "\"file-%s\" -> \"task-%d\";\n", m->file->cached_name, id);
+		fprintf(q->graph_logfile, "%s ", m->file->cached_name);
 	}
 
+	fprintf(q->graph_logfile, "OUTPUTS ");
+
 	LIST_ITERATE(t->output_mounts, m)
 	{
-		fprintf(q->graph_logfile, "\"task-%d\" -> \"file-%s\";\n", id, m->file->cached_name);
+		fprintf(q->graph_logfile, "%s ", m->file->cached_name);
 	}
+
+	fprintf(q->graph_logfile, "\n");
 }
 
 void vine_taskgraph_log_write_mini_task(struct vine_manager *q, struct vine_task *t, const char *task_name, const char *output_name)
@@ -58,31 +59,35 @@ void vine_taskgraph_log_write_mini_task(struct vine_manager *q, struct vine_task
 	if (p)
 		*p = 0;
 
-	fprintf(q->graph_logfile, "\"task-%d\" [color=green,label=\"%s\"];\n", id, show_names ? task_name : "");
+	fprintf(q->graph_logfile, "TASK T-%d \"%s\" INPUTS ", id, task_name);
 
 	free(name);
 
 	struct vine_mount *m;
 
 	LIST_ITERATE(t->input_mounts, m)
 	{
-		fprintf(q->graph_logfile, "\"file-%s\" -> \"task-%d\";\n", m->file->cached_name, id);
+		fprintf(q->graph_logfile, "%s ", m->file->cached_name);
 	}
 
-	/* A mini-task has one implied output that is named by provided argument, not the data structure */
-	fprintf(q->graph_logfile, "\"task-%d\" -> \"file-%s\";\n", id, output_name);
+	/* A mini-task has exactly one implied output that is named by provided argument, not the data structure */
+	fprintf(q->graph_logfile, "OUTPUTS %s", output_name);
+
+	fprintf(q->graph_logfile, "\n");
 }
 
 void vine_taskgraph_log_write_file(struct vine_manager *q, struct vine_file *f)
 {
 	if (!f)
 		return;
 
-	fprintf(q->graph_logfile, "\"file-%s\" [shape=rect,color=blue,label=\"%s\"];\n", f->cached_name, (show_names && f->source) ? path_basename(f->source) : "");
+	fprintf(q->graph_logfile, "FILE %s \"%s\" %ld\n", f->cached_name, f->source ? f->source : "", f->size);
+
+	/* If this file was generated by a mini-task, then describe that too. */
 	vine_taskgraph_log_write_mini_task(q, f->mini_task, f->source, f->cached_name);
 }
 
 void vine_taskgraph_log_write_footer(struct vine_manager *q)
 {
-	fprintf(q->graph_logfile, "}\n");
+	fprintf(q->graph_logfile, "# end\n");
 }

taskvine/src/tools/Makefile

@@ -5,7 +5,7 @@ LOCAL_LINKAGE+=${CCTOOLS_HOME}/taskvine/src/manager/libtaskvine.a ${CCTOOLS_HOME
 LOCAL_CCFLAGS+=-I ${CCTOOLS_HOME}/taskvine/src/manager
 
 PROGRAMS = vine_status vine_benchmark
-SCRIPTS = vine_graph_log vine_graph_workers vine_plot_txn_log vine_profile_dispatch vine_submit_workers vine_transfer_plot_animate vine_plot_compose
+SCRIPTS = vine_graph_log vine_plot_taskgraph vine_graph_workers vine_plot_txn_log vine_profile_dispatch vine_submit_workers vine_transfer_plot_animate vine_plot_compose
 TEST_PROGRAMS = vine_test
 TARGETS = $(PROGRAMS) $(TEST_PROGRAMS)