Skip to content

Commit 5ac33d2

Browse files
shnizzedyshnizzedy
committed
📝 Document changes in pipeline config for outputs
1 parent 8868981 commit 5ac33d2

File tree

2 files changed

+265
-20
lines changed

2 files changed

+265
-20
lines changed

‎docs/_sources/user/output_config.rst‎

Lines changed: 88 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -36,23 +36,91 @@ Configuration Without the GUI
3636

3737
The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run:
3838

39-
.. csv-table::
40-
:header: "Key","Description","Potential Values"
41-
:widths: 5,30,15
42-
:file: ../_static/params/output_config.csv
43-
44-
The box below contains an example of what these parameters might look like when defined in the YAML::
45-
46-
pipelineName : gelastic_epilepsy_pipeline
47-
workingDirectory : /data/working
48-
crashLogDirectory : /data/crash
49-
logDirectory : /data/logs
50-
outputDirectory : /data/output
51-
awsOutputBucketCredentials : /data/credentials.csv
52-
s3Encryption : [1]
53-
generateQualityControlImages : [1]
54-
write_func_outputs : [0]
55-
write_debugging_outputs : [0]
56-
removeWorkingDir : False
57-
reGenerateOutputs : False
58-
runSymbolicLinks : [1]
39+
.. table::
40+
:widths: 20,30,15
41+
:align: left
42+
43+
+------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
44+
| Key | Description | Potential Values |
45+
+==================================================================================================================+============================================================================================================================================================================+================================================================================================+
46+
| FROM | A pipeline configuration to base this pipeline on, for inferring unspecified configuration options. | A path (e.g., ``/data/another_analysis/pipeline.yml``) or the name of a preconfigured pipeline |
47+
+----------------+-------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
48+
| pipeline_setup | pipeline_name | A name that you would like to give to this pipeline. | A string. |
49+
| +---------------------+---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
50+
| | output_directory | path | The output directory for the pipeline. | A path (e.g., ``/data/my_analysis/output``). |
51+
| | +---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
52+
| | | write_func_outputs | Include extra versions and intermediate steps of functional preprocessing in the output directory. | On, Off, True, False |
53+
| | +---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
54+
| | | write_debugging_outputs | Include extra outputs in the output directory that may be of interest when more information is needed. | On, Off, True, False |
55+
| | +---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
56+
| | | output_tree | Output directory format and structure. | default, ndmg |
57+
| | +---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
58+
| | | generate_quality_control_images | Generate quality control pages containing preprocessing and derivative outputs. | On, Off, True, False |
59+
| +---------------------+---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
60+
| | working_directory | path | The working directory to be used by the pipeline during the run. | A path (e.g., ``/data/my_analysis/working``). |
61+
| | +---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
62+
| | | remove_working_dir | Deletes the contents of the Working Directory after running. This saves disk space, but any additional preprocessing or analysis will have to be completely re-run. | On, Off, True, False |
63+
| +---------------------+---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
64+
| | log_directory | path | Directory where CPAC should place run logs. | A path (e.g., ``/data/my_analysis/logs``). |
65+
| | +---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
66+
| | | run_logging | Whether to write log details of the pipeline run to the logging files. | On, Off, True, False |
67+
| +---------------------+---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
68+
| | crash_log_directory | path | The directory where C-PAC will store crash logs. | A path (e.g., ``/data/my_analysis/crash``). |
69+
| +---------------------+---------------------------------+-----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
70+
| | system_config | on_grid | run | Select Off if you intend to run CPAC on a single machine. If set to On, CPAC will attempt to submit jobs through the job scheduler / resource manager selected below. | On, Off, True, False |
71+
| | | +-----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
72+
| | | | resource_manager | Sun Grid Engine (SGE), Portable Batch System (PBS), or Simple Linux Utility for Resource Management (SLURM). Only applies if you are running on a grid or compute cluster. | SGE, PBS, SLURM |
73+
| | | +-----+-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
74+
| | | | SGE | parallel_environment | SGE Parallel Environment to use when running CPAC. Only applies when you are running on a grid or compute cluster using SGE. | A string. |
75+
| | | | +-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
76+
| | | | | queue | SGE Queue to use when running CPAC. Only applies when you are running on a grid or compute cluster using SGE. | A string. |
77+
| | +---------------------------------+-----+-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
78+
| | | maximum_memory_per_participant | The maximum amount of memory each participant's workflow can allocate. | A number (GB). |
79+
| | +---------------------------------+-----+-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
80+
| | | max_cores_per_participant | The maximum amount of cores (on a single machine) or slots on a node (on a cluster/grid) to allocate per participant. | An integer. |
81+
| | +---------------------------------+-----+-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
82+
| | | num_ants_threads | The number of cores to allocate to ANTS-based anatomical registration per participant. | An integer. |
83+
| | +---------------------------------+-----+-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
84+
| | | num_participants_at_once | The number of participant workflows to run at the same time. | An integer. |
85+
| +---------------------+---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
86+
| | Amazon-AWS | aws_output_bucket_credentials | Path to a set of AWS credentials if your output directory is set to an S3 bucket. | A path (e.g., ``/data/my_analysis/credentials.csv``). |
87+
| | +---------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
88+
| | | s3_encryption | Enable server-side 256-AES encryption on data to the S3 bucket. | An integer. |
89+
+----------------+---------------------+---------------------------------+-----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
90+
91+
92+
The box below contains an example of what these parameters might look like when defined in the YAML
93+
94+
.. code-block:: YAML
95+
96+
FROM: default
97+
pipeline_setup:
98+
pipeline_name: cpac-default-pipeline
99+
output_directory:
100+
path: /output
101+
write_func_outputs: False
102+
write_debugging_outputs: False
103+
output_tree: default
104+
generate_quality_control_images: True
105+
working_directory:
106+
path: /tmp
107+
remove_working_dir: True
108+
log_directory:
109+
path: /logs
110+
run_logging: True
111+
crash_log_directory:
112+
path: /crash
113+
system_config:
114+
on_grid:
115+
run: Off
116+
resource_manager: SGE
117+
SGE:
118+
parallel_environment: mpi_smp
119+
queue: all.q
120+
maximum_memory_per_participant: 1
121+
max_cores_per_participant: 1
122+
num_ants_threads: 1
123+
num_participants_at_once: 1
124+
Amazon-AWS:
125+
aws_output_bucket_credentials:
126+
s3_encryption: False

0 commit comments

Comments
 (0)