Skip to content

Commit 6a82b2b

Browse files
authored
📝 Document mem_x (#253)
2 parents 3784026 + 997b2bd commit 6a82b2b

File tree

1 file changed

+24
-16
lines changed

1 file changed

+24
-16
lines changed

docs/_sources/developer/nodes.rst

Lines changed: 24 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,23 @@ Nodes
88

99
.. _n_cpus:
1010

11-
.. warning::
12-
A Nipype :py:class:`~nipype.pipeline.engine.nodes.Node` has an initialization parameter ``mem_gb`` that differs from the :doc:`commandline option </user/run/help>` ``--mem_gb``. While the commandline option is **a limit**, the Node initialization parameter is **an estimate** of the most memory that Node will consume when run. The Node parameter is not a limit; rather, this value is used to allocate system resources at runtime.
13-
14-
Conversely, the commandline option ``--n_cpus`` is **a limit** and the Node initialization parameter ``n_procs`` is **also a limit** of the maximum number of threads a Node will be permmitted to consume.
15-
16-
C-PAC automatically creates a JSON-like file called ``callback.log`` (via the function :py:func:`~CPAC.utils.monitoring.log_nodes_cb`) when running. This file includes for each Node:
17-
18-
* estimated memory,
19-
* memory usage observed at runtime,
20-
* specified maximum number of threads per Node, and
21-
* threads used at runtime.
22-
23-
The Nipype utility function :py:func:`~nipype.utils.draw_gantt_chart.log_to_dict` reads a log file generated by ``log_nodes_cb`` to a Python dictionary.
24-
25-
When a developer creates or modifies a Node in C-PAC, a ``mem_gb`` and ``n_procs`` argument should be provided unless the respective defaults of 0.2 and None (number of available system cores) are expected to be sufficient. When testing, the ``mem_gb`` and ``n_procs`` arguments should be adjusted if the observed memory and/or thread usage of a Node exceeds the estimate.
26-
11+
A Nipype :py:class:`~nipype.pipeline.engine.nodes.Node` has an initialization parameter ``mem_gb`` that differs from the :doc:`commandline option </user/run/help>` ``--mem_gb``. While the commandline option is **a limit**, the Node initialization parameter is **an estimate** of the most memory that Node will consume when run. The Node parameter is not a limit; rather, this value is used to allocate system resources at runtime.
12+
13+
Conversely, the commandline option ``--n_cpus`` is **a limit** and the Node initialization parameter ``n_procs`` is **also a limit** of the maximum number of threads a Node will be permmitted to consume.
14+
15+
C-PAC automatically creates a JSON-like file called ``callback.log`` (via the function :py:func:`~CPAC.utils.monitoring.log_nodes_cb`) when running. This file includes for each Node:
16+
17+
* estimated memory,
18+
* memory usage observed at runtime,
19+
* specified maximum number of threads per Node, and
20+
* threads used at runtime.
21+
22+
When a developer creates or modifies a Node in C-PAC, a ``mem_gb`` and ``n_procs`` argument should be provided unless the respective defaults of 0.2 and None (number of available system cores) are expected to be sufficient. When testing, the ``mem_gb`` and ``n_procs`` arguments should be adjusted if the observed memory and/or thread usage of a Node exceeds the estimate.
23+
24+
For nodes that will use a varying amount of memory depending on the node's input data, the optional parameter ``mem_x`` takes a tuple of ``(multiplier, multiplicand)`` where ``multiplier`` is a number and ``multiplicand`` is the string name of the input to multiply such that the memory estimate returned by the ``mem_gb`` attribute is the ``mem_gb`` argument plus ``multiplier`` times ``x`` × ``y`` × ``z`` × ``t`` of the ``multiplicand`` file.
25+
26+
.. note::
27+
``mem_x`` is a new parameter in C-PAC v1.8.1 and subject to change in future releases as we continue to `develop methods for setting data- and operation-dependent memory estimates <https://github.com/FCP-INDI/C-PAC/issues/1509>`_.
2728

2829
.. autoclass:: CPAC.pipeline.nipype_pipeline_engine.Node
2930
:special-members: __init__
@@ -35,6 +36,13 @@ Nodes
3536
:members:
3637
:inherited-members:
3738

39+
.. autoclass:: CPAC.pipeline.nipype_pipeline_engine.Workflow
40+
:special-members: __init__
41+
:members:
42+
:inherited-members:
43+
44+
The Nipype utility function :py:func:`~nipype.utils.draw_gantt_chart.log_to_dict` reads a log file generated by ``log_nodes_cb`` to a Python dictionary.
45+
3846
.. autofunction:: CPAC.utils.monitoring.log_nodes_cb
3947

4048
draw_gantt_chart

0 commit comments

Comments
 (0)