You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/_sources/developer/nodes.rst
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,9 +7,9 @@ Nodes
7
7
8
8
.. _n_cpus:
9
9
10
-
A Nipype :py:class:`~nipype.pipeline.engine.nodes.Node` has an initialization parameter ``mem_gb`` that differs from the :doc:`commandline option </user/run/help>` ``--mem_gb``. While the commandline option is **a limit**, the Node initialization parameter is **an estimate** of the most memory that Node will consume when run. The Node parameter is not a limit; rather, this value is used to allocate system resources at runtime.
10
+
A Nipype :py:class:`~nipype.pipeline.engine.nodes.Node` has an initialization parameter ``mem_gb`` that differs from the :doc:`commandline option </user/run/help>` ``--mem-gb``. While the commandline option is **a limit**, the Node initialization parameter is **an estimate** of the most memory that Node will consume when run. The Node parameter is not a limit; rather, this value is used to allocate system resources at runtime.
11
11
12
-
Conversely, the commandline option ``--n_cpus`` is **a limit** and the Node initialization parameter ``n_procs`` is **also a limit** of the maximum number of threads a Node will be permmitted to consume.
12
+
Conversely, the commandline option ``--n-cpus`` is **a limit** and the Node initialization parameter ``n_procs`` is **also a limit** of the maximum number of threads a Node will be permmitted to consume.
13
13
14
14
C-PAC automatically creates a JSON-like file called ``callback.log`` (via the function :py:func:`~CPAC.utils.monitoring.log_nodes_cb`) when running. This file includes for each Node:
15
15
@@ -18,7 +18,7 @@ C-PAC automatically creates a JSON-like file called ``callback.log`` (via the fu
18
18
* specified maximum number of threads per Node, and
19
19
* threads used at runtime.
20
20
21
-
A ``callback.log`` can be provided to the pipeline configuration file (see :doc:`/user/compute_config`) or with the commandline flag ``--runtime_usage``. If a callback log is provided in the pipeline configuration, nodes with names that match nodes recorded in that pipeline log will have their memory estimates overridden by the values in the callback log plus a buffer percent (provided with the ``--runtime_buffer`` flag or in the pipeline configuration file).
21
+
A ``callback.log`` can be provided to the pipeline configuration file (see :doc:`/user/compute_config`) or with the commandline flag ``--runtime-usage``. If a callback log is provided in the pipeline configuration, nodes with names that match nodes recorded in that pipeline log will have their memory estimates overridden by the values in the callback log plus a buffer percent (provided with the ``--runtime-buffer`` flag or in the pipeline configuration file).
22
22
23
23
When a developer creates or modifies a Node in C-PAC, a ``mem_gb`` and ``n_procs`` argument should be provided unless the respective defaults of 0.2 and None (number of available system cores) are expected to be sufficient. When testing, the ``mem_gb`` and ``n_procs`` arguments should be adjusted if the observed memory and/or thread usage of a Node exceeds the estimate.
#. **Callback log - [text]:** The path to a callback log file from a previous run, including any resource-management parameters that will be applied in this run, like ``n_cpus`` and ``num_ants_threads``. This file is used override memory estimates with previously observed memory usage. Can be overridden with the commandline flag ``--runtime_usage``.
24
-
#. **Buffer - [percent]:** A percent of the previously observed memory usage that is to be added to the memory estimate. Default: 10. Can be overridden with the commandline flag ``--runtime_buffer``.
23
+
#. **Callback log - [text]:** The path to a callback log file from a previous run, including any resource-management parameters that will be applied in this run, like ``n_cpus`` and ``num_ants_threads``. This file is used override memory estimates with previously observed memory usage. Can be overridden with the commandline flag ``--runtime-usage``.
24
+
#. **Buffer - [percent]:** A percent of the previously observed memory usage that is to be added to the memory estimate. Default: 10. Can be overridden with the commandline flag ``--runtime-buffer``.
25
25
26
26
#. **Number of Participants to Run Simultaneously - [integer]:** This number depends on computing resources.
Copy file name to clipboardExpand all lines: docs/_sources/user/cpac.rst
+32-8Lines changed: 32 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,12 +3,26 @@ cpac (Python package)
3
3
4
4
`cpac <https://pypi.org/project/cpac/>`_ is available so that you can easily run analyses without needing interact with the container platform that allows you to run C-PAC without installing all of the underlying software.
5
5
6
-
cpac requires Python 3.6 or greater. To get cpac, simply
6
+
``cpac`` requires Python 3.8 or greater. To get cpac, simply
7
7
8
8
.. code-block:: console
9
9
10
10
pip install cpac
11
11
12
+
Run extra packages with cpac
13
+
````````````````````````````
14
+
Beginning with v1.8.5, ``cpac`` can run |ba_timeseries_gradients|_ and/or |tsconcat|_ in addition to C-PAC. If either package is already installed, ``cpac`` will use the installed version. You can install them with the relevant extras, like one of the following lines:
``ba_timeseries_gradients`` and ``tsconcat`` may have more specific dependency requirements than C-PAC. If you run into unclear installation issues when installing these extras, please see the documentation for those packages.
25
+
12
26
Download / Upgrade C-PAC with cpac
13
27
``````````````````````````````````
14
28
@@ -88,32 +102,42 @@ To run C-PAC with a pipeline configuration file other than one of the pre-config
88
102
89
103
.. code-block:: console
90
104
91
-
cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant --pipeline_file /Users/You/Documents/pipeline_config.yml
105
+
cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant --pipeline-file /Users/You/Documents/pipeline_config.yml
92
106
93
107
Finally, to run C-PAC with a specific data configuration file (instead of providing a BIDS data directory):
94
108
95
109
.. code-block:: console
96
110
97
-
cpac run /Users/You/any_directory /Users/You/some_folder_for_outputs participant --data_config_file /Users/You/Documents/data_config.yml
111
+
cpac run /Users/You/any_directory /Users/You/some_folder_for_outputs participant --data-config-file /Users/You/Documents/data_config.yml
98
112
99
-
Note: we are still providing the postionally-required ``bids_dir`` input parameter. However C-PAC will not look for data in this directory when you provide a data configuration YAML with the ``--data_config_file`` flag. Providing ``.`` or ``$PWD`` will simply pass the present working directory. In addition, if the dataset in your data configuration file is not in BIDS format, just make sure to add the ``--skip_bids_validator`` flag at the end of your command to bypass the BIDS validation process.
113
+
Note: we are still providing the postionally-required ``bids_dir`` input parameter. However C-PAC will not look for data in this directory when you provide a data configuration YAML with the ``--data-config-file`` flag. Providing ``.`` or ``$PWD`` will simply pass the present working directory. In addition, if the dataset in your data configuration file is not in BIDS format, just make sure to add the ``--skip-bids-validator`` flag at the end of your command to bypass the BIDS validation process.
100
114
101
115
The full list of parameters and options that can be passed to C-PAC are shown below:
102
116
103
117
.. include:: /user/run/help.rst
104
118
105
119
.. include:: /user/utils/help.rst
106
120
107
-
Note that any of the optional arguments above will over-ride any pipeline settings in the default pipeline or in the pipeline configuration file you provide via the ``--pipeline_file`` parameter.
121
+
Note that any of the optional arguments above will over-ride any pipeline settings in the default pipeline or in the pipeline configuration file you provide via the ``--pipeline-file`` parameter.
108
122
109
123
**Further usage notes:**
110
124
111
-
* You can run only anatomical preprocessing easily, without modifying your data or pipeline configuration files, by providing the ``--anat_only`` flag.
125
+
* You can run only anatomical preprocessing easily, without modifying your data or pipeline configuration files, by providing the ``--anat-only`` flag.
112
126
113
-
* As stated, the default behavior is to read data that is organized in the BIDS format. This includes data that is in Amazon AWS S3 by using the format ``s3://<bucket_name>/<bids_dir>`` for the ``bids_dir`` command line argument. Outputs can be written to S3 using the same format for the ``output_dir``. Credentials for accessing these buckets can be specified on the command line (using ``--aws_input_creds`` or ``--aws_output_creds``).
127
+
* As stated, the default behavior is to read data that is organized in the BIDS format. This includes data that is in Amazon AWS S3 by using the format ``s3://<bucket_name>/<bids_dir>`` for the ``bids_dir`` command line argument. Outputs can be written to S3 using the same format for the ``output_dir``. Credentials for accessing these buckets can be specified on the command line (using ``--aws-input-creds`` or ``--aws-output-creds``).
114
128
115
-
* When the app is run, a data configuration file is written to the working directory. This directory can be specified with ``--working_dir`` or the directory from which you run ``cpac`` will be used. This file can be passed into subsequent runs, which avoids the overhead of re-parsing the BIDS input directory on each run (i.e. for cluster or cloud runs). These files can be generated without executing the C-PAC pipeline using the ``test_run`` command line argument.
129
+
* When the app is run, a data configuration file is written to the working directory. This directory can be specified with ``--working-dir`` or the directory from which you run ``cpac`` will be used. This file can be passed into subsequent runs, which avoids the overhead of re-parsing the BIDS input directory on each run (i.e. for cluster or cloud runs). These files can be generated without executing the C-PAC pipeline using ``test_config`` as the analysis level.
116
130
117
131
* The ``participant_label`` and ``participant_ndx`` arguments allow the user to specify which of the many datasets should be processed, which is useful when parallelizing the run of multiple participants.
118
132
119
133
* If you want to pass runtime options to your container plaform (Docker or Singularity), you can pass them with ``-o`` or ``--container_options``.
134
+
135
+
.. TODO: Update cpac to handle `-`s and `_`s like C-PAC
0 commit comments