📝 Document "✨ Add --runtime_usage flag (FCP-INDI/C-PAC#1701) (#276)

Author: shnizzedy

.circleci/config.yml

@@ -15,7 +15,7 @@ commands:
           command: |
             sudo apt-get update && sudo apt-get -y install git python3-dev graphviz graphviz-dev libgraphviz-dev pkg-config python3-sphinx
             pip install --user -r requirements.txt
-            pip install --user -r https://raw.githubusercontent.com/FCP-INDI/C-PAC/<< parameters.version >>/requirements.txt
+            pip install --user -r https://raw.githubusercontent.com/${CIRCLE_PROJECT_USERNAME}/C-PAC/<< parameters.version >>/requirements.txt
             pip install --user git+https://github.com/${CIRCLE_PROJECT_USERNAME}/C-PAC.git@<< parameters.version >>
             git clone https://github.com/${CIRCLE_PROJECT_USERNAME}/C-PAC.git /home/circleci/build/C-PAC
             cd /home/circleci/build/C-PAC

docs/_sources/_static/custom.css

@@ -21,6 +21,9 @@ div.bodywrapper section {
   background-color: white;
   padding: 0 20px 30px 20px;
 }
+div.bodywrapper section > p {
+  padding-bottom: 1ex;
+}
 div.bodywrapper section h2 {
   margin-block-start: 0;
   padding-block-start: 0.83em;

docs/_sources/developer/nodes.rst

@@ -18,6 +18,8 @@ C-PAC automatically creates a JSON-like file called ``callback.log`` (via the fu
    * specified maximum number of threads per Node, and
    * threads used at runtime.
 
+A ``callback.log`` can be provided to the pipeline configuration file (see :doc:`/user/compute_config`) or with the commandline flag ``--runtime_usage``. If a callback log is provided in the pipeline configuration, nodes with names that match nodes recorded in that pipeline log will have their memory estimates overridden by the values in the callback log plus a buffer percent (provided with the ``--runtime_buffer`` flag or in the pipeline configuration file).
+
 When a developer creates or modifies a Node in C-PAC, a ``mem_gb`` and ``n_procs`` argument should be provided unless the respective defaults of 0.2 and None (number of available system cores) are expected to be sufficient. When testing, the ``mem_gb`` and ``n_procs`` arguments should be adjusted if the observed memory and/or thread usage of a Node exceeds the estimate.
 
 For nodes that will use a varying amount of memory depending on the node's input data, the optional parameter ``mem_x`` takes a tuple of ``(memory multiplier, input file, multiplier mode)`` where ``memory multiplier`` is a number and ``input file`` is the string name of the Node input to multiply such that the memory estimate returned by the ``mem_gb`` attribute is the ``mem_gb`` argument plus ``memory multiplier`` times the dimensions of the ``input file`` as specified in the ``multiplier mode`` (``xyzt`` (spatial × temporal; default), ``xyz`` (spatial), or just ``t`` (temporal)).

docs/_sources/user/anat.rst

@@ -355,7 +355,7 @@ Configuring CPAC to Run Anatomical Tissue Segmentation
 
 #. **Erosion - [On, Off]:** Whether or not to use erosion to erode binarized tissue masks.
 
-#. **Erosion Proportion - [float]:** Set the erosion proportion, if use erosion to erode binarized tissue masks. The default is 0.6.
+#. **Erosion Proportion - [float]:** Set the target volume ratio, if using erosion to erode binarized tissue masks. The default is 0.6.
 
 .. figure:: /_images/seg_gui_2.png
 

docs/_sources/user/compute_config.rst

@@ -8,6 +8,12 @@ Computer Settings
 
 #. **Maximum Cores Per Participant - [integer]:** Number of cores (on a single machine) or slots on a node (cluster/grid) per subject. Slots are cores on a cluster/grid node. 'Number of Cores Per Participant' multiplied by 'Number of Participants to Run Simultaneously' must not be greater than the total number of cores. Dedicating more than one core/CPU per participant will direct C-PAC to parallelize the motion correction and time series registration transform application steps, for a speed increase.
 
+#. **Raise Insufficient - [True, False]:** If True (default), C-PAC will raise an error before trying to run a pipeline if it estimates the given data and pipeline configuration will exceed the memory or core limits. If False, C-PAC will raise a warning and continue to run the pipeline even if it estimates the given data and pipeline configuration will exceed the memory or core limits.
+
+#. **Observed Usage:** The resource usage of each node depends on many factors, including the data, the pipeline configuration, and the system. To run many subjects with optimimzed resource estimation, first run a single subject with the desired number of cores and with a generous memory limit. Then, provide the ``callback.log`` generated from that initial run when you run the rest of the subjects.
+    #. **Callback log - [text]:** The path to a callback log file from a previous run, including any resource-management parameters that will be applied in this run, like ``n_cpus`` and ``num_ants_threads``. This file is used override memory estimates with previously observed memory usage. Can be overridden with the commandline flag ``--runtime_usage``.
+    #. **Buffer - [percent]:** A percent of the previously observed memory usage that is to be added to the memory estimate. Default: 10. Can be overridden with the commandline flag ``--runtime_buffer``.
+
 #. **Number of Participants to Run Simultaneously - [integer]:** This number depends on computing resources.
 
 #. **Number of Cores for Anatomical Registration (ANTS) - [integer]:** This number depends on computing resources.