@@ -8,6 +8,7 @@ These features allows users to ensure high throughput of their data processing
8
8
while also controlling the amount of computing resources a given workflow will
9
9
use.
10
10
11
+
11
12
Specifying Resources in the Node Interface
12
13
==========================================
13
14
Each ``Node `` instance interface has two parameters that specify its expected
@@ -23,6 +24,7 @@ particular node is expected to use 8 threads and 2 GB of memory:
23
24
If the resource parameters are never set, they default to being 1 thread and 1
24
25
GB of RAM.
25
26
27
+
26
28
Resource Scheduler
27
29
==================
28
30
The ``MultiProc `` workflow plugin schedules node execution based on the
@@ -43,28 +45,29 @@ for ``n_procs`` and ``memory_gb``, respectively.
43
45
The plugin will then queue eligible nodes for execution based on their expected
44
46
usage via the ``num_threads `` and ``estimated_memory_gb `` interface parameters.
45
47
If the plugin sees that only 3 of its 4 processors and 4 GB of its 6 GB of RAM
46
- are being used, it will attempt to execute the next available node as long as
47
- its ``num_threads = 1 `` and ``estimated_memory_gb <= 2 ``. If this is not the
48
- case, it will continue to check every available node in the queue until it sees
49
- a node that meets these conditions or it waits for a executing node to finish to
50
- earn back the necessary resources. The priority of the queue is highest for
51
- nodes with the most ``estimated_memory_gb `` followed by nodes with the most
52
- expected ``num_threads ``.
48
+ are being used by running nodes, it will attempt to execute the next available
49
+ node as long as its ``num_threads = 1 `` and ``estimated_memory_gb <= 2 ``. If
50
+ this is not the case, it will continue to check every available node in the
51
+ queue until it sees a node that meets these conditions, or it waits for an
52
+ executing node to finish to earn back the necessary resources. The priority of
53
+ the queue is highest for nodes with the most ``estimated_memory_gb `` followed
54
+ by nodes with the most expected ``num_threads ``.
55
+
53
56
54
57
Runtime Profiler and using the Callback Log
55
58
===========================================
56
59
It is not always easy to estimate the amount of resources a particular function
57
60
or command uses. To help with this, Nipype provides some feedback about the
58
61
system resources used by every node during workflow execution via the built-in
59
- runtime profiler. The runtime profiler is automatically enabled if the]
62
+ runtime profiler. The runtime profiler is automatically enabled if the
60
63
``psutil `` Python package is installed and found on the system. If the package
61
64
is not found, the workflow will run normally without the runtime profiler.
62
65
63
66
The runtime profiler records the number of threads and the amount of memory (GB)
64
67
used as ``runtime_threads `` and ``runtime_memory_gb `` in the Node's
65
- ``result.runtime `` parameter . Since the node object is pickled and written to
68
+ ``result.runtime `` attribute . Since the node object is pickled and written to
66
69
disk in its working directory, these values are available for analysis after
67
- node or workflow execution by parsing the pickle file contents in Python .
70
+ node or workflow execution by manually parsing the pickle file contents.
68
71
69
72
Nipype also provides a logging mechanism for saving node runtime statistics to
70
73
a JSON-style log file via the ``log_nodes_cb `` logger function. This is enabled
@@ -88,7 +91,7 @@ configured.
88
91
handler = logging.FileHandler(callback_log_path)
89
92
logger.addHandler(handler)
90
93
91
- Finally, the workflow can be ran .
94
+ Finally, the workflow can be run .
92
95
93
96
::
94
97
workflow.run(plugin='MultiProc', plugin_args=args_dict)
@@ -112,6 +115,7 @@ the user can change the node interface ``num_threads`` and
112
115
``estimated_memory_gb `` parameters to reflect this for a higher pipeline
113
116
throughput.
114
117
118
+
115
119
Visualizing Pipeline Resources
116
120
==============================
117
121
Nipype provides the ability to visualize the workflow execution based on the
@@ -131,4 +135,9 @@ generated from the callback logger after workflow execution - as shown above.
131
135
# ...creates gantt chart in '/home/user/run_stats.log.html'
132
136
133
137
The `generate_gantt_chart`` function will create an html file that can be viewed
134
- in a browser.
138
+ in a browser. Below is an example of the gantt chart displayed in a web browser.
139
+ Note that when the cursor is hovered over any particular node bubble or resource
140
+ bubble, some additional information is shown in a pop-up.
141
+
142
+ * - .. image:: images/gantt_chart.png
143
+ :width: 100 %
0 commit comments