Skip to content

Commit e937bdc

Browse files
committed
Finished documentation with gantt chart image
1 parent de6a7f1 commit e937bdc

File tree

2 files changed

+21
-12
lines changed

2 files changed

+21
-12
lines changed

doc/images/gantt_chart.png

107 KB
Loading

doc/users/resource_sched_profiler.rst

Lines changed: 21 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ These features allows users to ensure high throughput of their data processing
88
while also controlling the amount of computing resources a given workflow will
99
use.
1010

11+
1112
Specifying Resources in the Node Interface
1213
==========================================
1314
Each ``Node`` instance interface has two parameters that specify its expected
@@ -23,6 +24,7 @@ particular node is expected to use 8 threads and 2 GB of memory:
2324
If the resource parameters are never set, they default to being 1 thread and 1
2425
GB of RAM.
2526

27+
2628
Resource Scheduler
2729
==================
2830
The ``MultiProc`` workflow plugin schedules node execution based on the
@@ -43,28 +45,29 @@ for ``n_procs`` and ``memory_gb``, respectively.
4345
The plugin will then queue eligible nodes for execution based on their expected
4446
usage via the ``num_threads`` and ``estimated_memory_gb`` interface parameters.
4547
If the plugin sees that only 3 of its 4 processors and 4 GB of its 6 GB of RAM
46-
are being used, it will attempt to execute the next available node as long as
47-
its ``num_threads = 1`` and ``estimated_memory_gb <= 2``. If this is not the
48-
case, it will continue to check every available node in the queue until it sees
49-
a node that meets these conditions or it waits for a executing node to finish to
50-
earn back the necessary resources. The priority of the queue is highest for
51-
nodes with the most ``estimated_memory_gb`` followed by nodes with the most
52-
expected ``num_threads``.
48+
are being used by running nodes, it will attempt to execute the next available
49+
node as long as its ``num_threads = 1`` and ``estimated_memory_gb <= 2``. If
50+
this is not the case, it will continue to check every available node in the
51+
queue until it sees a node that meets these conditions, or it waits for an
52+
executing node to finish to earn back the necessary resources. The priority of
53+
the queue is highest for nodes with the most ``estimated_memory_gb`` followed
54+
by nodes with the most expected ``num_threads``.
55+
5356

5457
Runtime Profiler and using the Callback Log
5558
===========================================
5659
It is not always easy to estimate the amount of resources a particular function
5760
or command uses. To help with this, Nipype provides some feedback about the
5861
system resources used by every node during workflow execution via the built-in
59-
runtime profiler. The runtime profiler is automatically enabled if the]
62+
runtime profiler. The runtime profiler is automatically enabled if the
6063
``psutil`` Python package is installed and found on the system. If the package
6164
is not found, the workflow will run normally without the runtime profiler.
6265

6366
The runtime profiler records the number of threads and the amount of memory (GB)
6467
used as ``runtime_threads`` and ``runtime_memory_gb`` in the Node's
65-
``result.runtime`` parameter. Since the node object is pickled and written to
68+
``result.runtime`` attribute. Since the node object is pickled and written to
6669
disk in its working directory, these values are available for analysis after
67-
node or workflow execution by parsing the pickle file contents in Python.
70+
node or workflow execution by manually parsing the pickle file contents.
6871

6972
Nipype also provides a logging mechanism for saving node runtime statistics to
7073
a JSON-style log file via the ``log_nodes_cb`` logger function. This is enabled
@@ -88,7 +91,7 @@ configured.
8891
handler = logging.FileHandler(callback_log_path)
8992
logger.addHandler(handler)
9093

91-
Finally, the workflow can be ran.
94+
Finally, the workflow can be run.
9295

9396
::
9497
workflow.run(plugin='MultiProc', plugin_args=args_dict)
@@ -112,6 +115,7 @@ the user can change the node interface ``num_threads`` and
112115
``estimated_memory_gb`` parameters to reflect this for a higher pipeline
113116
throughput.
114117

118+
115119
Visualizing Pipeline Resources
116120
==============================
117121
Nipype provides the ability to visualize the workflow execution based on the
@@ -131,4 +135,9 @@ generated from the callback logger after workflow execution - as shown above.
131135
# ...creates gantt chart in '/home/user/run_stats.log.html'
132136

133137
The `generate_gantt_chart`` function will create an html file that can be viewed
134-
in a browser.
138+
in a browser. Below is an example of the gantt chart displayed in a web browser.
139+
Note that when the cursor is hovered over any particular node bubble or resource
140+
bubble, some additional information is shown in a pop-up.
141+
142+
* - .. image:: images/gantt_chart.png
143+
:width: 100 %

0 commit comments

Comments
 (0)