Vine: Plotting tools cleanup (#4245)

* Add description of taskgraph file format

* Modify taskgraph log to generate abstract format, rather than writing graphviz dot directly.

* Add vine_plot_taskgraph to consume new taskgraph format.

* Organize plotting section to present each tool consistently:
debug, performance, taskgraph, transactions.

* Debug log first.

* format

* vine_graph_log -> vine_plot_performance
vine_graph_workers -> vine_plot_workers

* vine_graph_log -> vine_plot_performance

* Modify vine_plot_performance to generate (nearly) same output using matplotlib.

* format

* vine_graph_log -> vine_plot_performance

* vine_graph_log -> vine_plot_performance

* Rebuild all md files from source.  (Some changes propagated from prior commits.)

* Add matplotlib to developer environment.

* - Remove markers from points, since they all overlap.
- Make time resolution adaptive so that you get a reasonable number of points.

* remove changes to list.c

---------

Co-authored-by: Benjamin Tovar <[email protected]>

Author: dthain

doc/man/m4/vine_graph_log.m4 renamed to doc/man/m4/vine_plot_performance.m4

@@ -1,20 +1,19 @@
 include(manual.h)dnl
-HEADER(vine_graph_log)
+HEADER(vine_plot_performance)
 
 SECTION(NAME)
-BOLD(vine_graph_log) - plots TaskVine performance logs.
+BOLD(vine_plot_performance) - plots TaskVine performance logs.
 
 SECTION(SYNOPSIS)
-CODE(vine_graph_log [options] vine-perf-log)
+CODE(vine_plot_performance [options] vine-perf-log)
 
 SECTION(DESCRIPTION)
 
-BOLD(vine_graph_log) is a script to plot some of the statistics from a
+BOLD(vine_plot_performance) is a script to plot some of the statistics from a
 TaskVine log, as generated by CODE(vine_enable_perf_log) from the C API, or by
-CODE(q.enable_perf_log) from the Python API. It assumes the availability of
-CODE(gnuplot).
+CODE(q.enable_perf_log) from the Python API.
 
-BOLD(vine_graph_log) generates four plots, all with CODE(timestamp) as
+BOLD(vine_plot_performance) generates four plots, all with CODE(timestamp) as
 the independent variable:
 
 CODE(time):
@@ -33,9 +32,8 @@ SECTION(OPTIONS)
 OPTIONS_BEGIN
 OPTION_ARG_SHORT(o,prefix-output)Generate prefix-output.{time,time-log,tasks,tasks-log}.PARAM(output-format). Default is PARAM(work-queue-log).
 OPTION_ARG_SHORT(o,prefix-output)Generate prefix-output.{time,time-log,tasks,tasks-log}.PARAM(output-format). Default is PARAM(work-queue-log).
-OPTION_ARG_SHORT(c,gnuplot-path)Specify the location of the gnuplot executable. Default is gnuplot.
 OPTION_ARG_SHORT(r,range)Range of time to plot, in time units (see -u) from the start of execution. Of the form: min:max, min:, or :max.
-OPTION_ARG_SHORT(T,output-format)Set output format. Default is png. If "text", then the gnuplot scripts are written instead of the images.
+OPTION_ARG_SHORT(T,output-format)Set output format. Default is svg.
 OPTION_ARG_SHORT(u,time-unit)Time scale to output. One of s,m,h or d, for seconds, minutes (default), hours or days.
 OPTION_FLAG_SHORT(h)Show help text.
 OPTIONS_END
@@ -45,37 +43,21 @@ SECTION(EXAMPLES)
 General use:
 
 LONGCODE_BEGIN
-% vine_graph_log mylog
+% vine_plot_performance mylog
 % ls mylog*.png
 mylog.tasks.png mylog.tasks-log.png mylog.time.png mylog.time-log.png
 LONGCODE_END
 
 Plot up to the eleventh hour:
 
 LONGCODE_BEGIN
-% vine_graph_log -u h -r :11 mylog
+% vine_plot_performance -u h -r :11 mylog
 
 LONGCODE_END
 Generate script text:
 
-LONGCODE_BEGIN
-% vine_graph_log -Ttext mylog
-% ls mylog*.gnuplot
-mylog.tasks.png.gnuplot mylog.tasks-log.png.gnuplot mylog.time.png.gnuplot mylog.time-log.png.gnuplot
-LONGCODE_END
-
-Specify gnuplot path:
-
-LONGCODE_BEGIN
-% vine_graph_log -c/some/dir/bin/gnuplot mylog
-LONGCODE_END
-
 SECTION(BUGS)
 
-LIST_BEGIN
-LIST_ITEM(Some formats need a special setup for their gnuplot output terminal. `-T' blindly passes the output format, which may cause gnuplot to fail.)
-LIST_END
-
 SECTION(COPYRIGHT)
 
 COPYRIGHT_BOILERPLATE

doc/man/manual.h

@@ -22,7 +22,7 @@ define(SEE_ALSO_TASK_VINE,
 `LIST_BEGIN
 LIST_ITEM(MANUAL(Cooperative Computing Tools Documentation,"../index.html"))
 LIST_ITEM(MANUAL(TaskVine User Manual,"../taskvine.html"))
-LIST_ITEM(MANPAGE(vine_worker,1) MANPAGE(vine_status,1) MANPAGE(vine_factory,1) MANPAGE(vine_graph_log,1) )
+LIST_ITEM(MANPAGE(vine_worker,1) MANPAGE(vine_status,1) MANPAGE(vine_factory,1) MANPAGE(vine_plot_performance,1) )
 LIST_END')dnl
 dnl
 define(SEE_ALSO_PARROT,

doc/man/md/condor_submit_workers.md

@@ -101,7 +101,7 @@ The Cooperative Computing Tools are Copyright (C) 2022 The University of Notre D
 
 - [Cooperative Computing Tools Documentation]("../index.html")
 - [Work Queue User Manual]("../workqueue.html")
-- [work_queue_worker(1)](work_queue_worker.md) [work_queue_status(1)](work_queue_status.md) [work_queue_factory(1)](work_queue_factory.md) [condor_submit_workers(1)](condor_submit_workers.md) [sge_submit_workers(1)](sge_submit_workers.md) [torque_submit_workers(1)](torque_submit_workers.md) 
+- [work_queue_worker(1)](work_queue_worker.md) [work_queue_status(1)](work_queue_status.md) [work_queue_factory(1)](work_queue_factory.md) [condor_submit_workers(1)](condor_submit_workers.md) [uge_submit_workers(1)](uge_submit_workers.md) [torque_submit_workers(1)](torque_submit_workers.md) 
 
 
 CCTools

doc/man/md/makeflow.md

@@ -89,7 +89,7 @@ OPTION_END
 - **--send-environment**<br />Send all local environment variables in remote execution.
 - **--wait-for-files-upto=_&lt;#&gt;_**<br />Wait for output files to be created upto this many seconds (e.g., to deal with NFS semantics).
 - **-S**,**--submission-timeout=_&lt;timeout&gt;_**<br />Time to retry failed batch job submission. (default is 3600s)
-- **-T**,**--batch-type=_&lt;type&gt;_**<br />Batch system type: local, dryrun, condor, sge, pbs, torque, blue_waters, slurm, moab, cluster, wq, amazon, mesos. (default is local)
+- **-T**,**--batch-type=_&lt;type&gt;_**<br />Batch system type: local, dryrun, condor, wq, vine, uge, pbs, torque, slurm, moab, cluster, amazon. (default is local)
 - **--safe-submit-mode**<br />Excludes resources at submission. (SLURM, TORQUE, and PBS)
 - **--ignore-memory-spec**<br />Excludes memory at submission. (SLURM)
 - **--batch-mem-type=_&lt;type&gt;_**<br />Specify memory resource type. (SGE)

doc/man/md/makeflow_amazon_batch_cleanup.md

@@ -85,7 +85,7 @@ The Cooperative Computing Tools are Copyright (C) 2022 The University of Notre D
 
 - [Cooperative Computing Tools Documentation]("../index.html")
 - [Work Queue User Manual]("../workqueue.html")
-- [work_queue_worker(1)](work_queue_worker.md) [work_queue_status(1)](work_queue_status.md) [work_queue_factory(1)](work_queue_factory.md) [condor_submit_workers(1)](condor_submit_workers.md) [sge_submit_workers(1)](sge_submit_workers.md) [torque_submit_workers(1)](torque_submit_workers.md) 
+- [work_queue_worker(1)](work_queue_worker.md) [work_queue_status(1)](work_queue_status.md) [work_queue_factory(1)](work_queue_factory.md) [condor_submit_workers(1)](condor_submit_workers.md) [uge_submit_workers(1)](uge_submit_workers.md) [torque_submit_workers(1)](torque_submit_workers.md) 
 
 
 CCTools

doc/man/md/makeflow_amazon_batch_setup.md

@@ -90,7 +90,7 @@ The Cooperative Computing Tools are Copyright (C) 2022 The University of Notre D
 
 - [Cooperative Computing Tools Documentation]("../index.html")
 - [Work Queue User Manual]("../workqueue.html")
-- [work_queue_worker(1)](work_queue_worker.md) [work_queue_status(1)](work_queue_status.md) [work_queue_factory(1)](work_queue_factory.md) [condor_submit_workers(1)](condor_submit_workers.md) [sge_submit_workers(1)](sge_submit_workers.md) [torque_submit_workers(1)](torque_submit_workers.md) 
+- [work_queue_worker(1)](work_queue_worker.md) [work_queue_status(1)](work_queue_status.md) [work_queue_factory(1)](work_queue_factory.md) [condor_submit_workers(1)](condor_submit_workers.md) [uge_submit_workers(1)](uge_submit_workers.md) [torque_submit_workers(1)](torque_submit_workers.md) 
 
 
 CCTools