📝 Add documentation about nested pipelines and importing default values

shnizzedy · shnizzedy · commit 0dcc2ca77ab4 · 2021-01-13T15:34:32.000-05:00
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -28,7 +28,9 @@ commands:
     steps:
       - run:
           name: 🖇️ Linking references
-          command: ln $(python -c "import os, CPAC; print(os.path.abspath(os.path.dirname(CPAC.__file__)))")/resources/configs/1.7-1.8-nesting-mappings.yml docs/_sources/references/1.7-1.8-nesting-mappings.yml || true
+          command: |
+            ln $(python -c "import os, CPAC; print(os.path.abspath(os.path.dirname(CPAC.__file__)))")/resources/configs/1.7-1.8-nesting-mappings.yml docs/_sources/references/1.7-1.8-nesting-mappings.yml || true
+            ln $(python -c "import os, CPAC; print(os.path.abspath(os.path.join(os.path.dirname(CPAC.__file__), os.pardir)))")/dev/docker_data/default_pipeline.yml docs/_sources/references/default_pipeline.yml || true
   run-cpac-commands:
     steps:
       - setup_remote_docker
diff --git a/docs/_sources/user/after_warp.rst b/docs/_sources/user/after_warp.rst
@@ -38,7 +38,7 @@ Configuring C-PAC to Run After-Warping Steps
 Configuration Using a YAML File
 """"""""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run the after warping options:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run the after warping options:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/alff.rst b/docs/_sources/user/alff.rst
@@ -40,7 +40,7 @@ Configuring CPAC to Run ALFF and f/ALFF
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run anatomical preprocessing:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run anatomical preprocessing:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/anat.rst b/docs/_sources/user/anat.rst
@@ -190,7 +190,7 @@ Configuring CPAC to Run Anatomical Registration
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run anatomical preprocessing:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run anatomical preprocessing:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
@@ -362,7 +362,7 @@ Configuring CPAC to Run Anatomical Tissue Segmentation
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run anatomical tissue segmentation:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run anatomical tissue segmentation:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/aroma_ica.rst b/docs/_sources/user/aroma_ica.rst
@@ -36,7 +36,7 @@ CONFIGURING C-PAC TO RUN AROMA-ICA:
 CONFIGURING AROMA-ICA USING A YAML FILE:
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run AROMA-ICA::
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run AROMA-ICA::
 
     runICA :  [1]
     aroma_TR :  0.01
diff --git a/docs/_sources/user/centrality.rst b/docs/_sources/user/centrality.rst
@@ -60,7 +60,7 @@ Note that only centrality measures that have binarized/weighted checked on this
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run anatomical preprocessing:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run anatomical preprocessing:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/compute_config.rst b/docs/_sources/user/compute_config.rst
@@ -27,7 +27,7 @@ Computer Settings
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/derivatives.rst b/docs/_sources/user/derivatives.rst
@@ -1,7 +1,7 @@
 Computable Derivatives
 =======================
 
-The following index provides links to pages containing overviews and configuration instructions for the various outputs that C-PAC is capable of computing.  For more information about how to set up a pipeline configuration (including preprocessing options), see the :doc:`previous section </user/pipeline_config>`.
+The following index provides links to pages containing overviews and configuration instructions for the various outputs that C-PAC is capable of computing.  For more information about how to set up a pipeline configuration (including preprocessing options), see the :doc:`previous section </user/pipelines/pipeline_config>`.
 
 .. toctree::
    :maxdepth: 1
diff --git a/docs/_sources/user/func.rst b/docs/_sources/user/func.rst
@@ -31,7 +31,7 @@ Initial Preprocessing
 
 Configuration Without the GUI
 """""""""""""""""""""""""""""
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run initial functional preprocessing:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run initial functional preprocessing:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
@@ -75,7 +75,7 @@ Note that if a scan parameters file was used to construct the participant list,
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run slice timing correction and drop TRs:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run slice timing correction and drop TRs:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
@@ -170,7 +170,7 @@ The C-PAC pipeline configuration builder provides options for configuring the Di
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run distortion correction:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run distortion correction:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
@@ -242,7 +242,7 @@ Configuring FSL BET options:
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run functional to anatomical registration:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run functional to anatomical registration:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
@@ -285,7 +285,7 @@ Functional to Template Registration
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run functional to anatomical registration:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run functional to anatomical registration:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/index.rst b/docs/_sources/user/index.rst
@@ -101,7 +101,7 @@ User Guide Index
 
    1. C-PAC Quickstart <quick>
    2. Specify Your Data <subject_list_config>
-   3. Select Your Pipeline <pipeline_config>
+   3. Select Your Pipeline <pipelines/pipeline_config>
    4. Pre-Process Your Data <preprocessing>
    5. Compute Derivatives <derivatives>
    6. All Run Options <running>
diff --git a/docs/_sources/user/longitudinal.rst b/docs/_sources/user/longitudinal.rst
@@ -24,7 +24,7 @@ Configuring CPAC to Run Longitudinal Preprocessing Pipeline
 
 Configuration Without the GUI
 """""""""""""""""""""""""""""
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run longitudinal preprocessing:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run longitudinal preprocessing:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/nuisance.rst b/docs/_sources/user/nuisance.rst
@@ -123,7 +123,7 @@ Configuring Nuisance Signal Regression Options
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run nuisance correction:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run nuisance correction:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
@@ -307,7 +307,7 @@ Configuring Median Angle Correction Options
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run median angle correction:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run median angle correction:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
@@ -331,7 +331,7 @@ Configuring Temporal Filtering Options
 Configuration Without the GUI
 """""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run temporal filtering:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run temporal filtering:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/output_config.rst b/docs/_sources/user/output_config.rst
@@ -34,7 +34,7 @@ Output Settings
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/pipelines/1.7-1.8-nesting-mappings.rst b/docs/_sources/user/pipelines/1.7-1.8-nesting-mappings.rst
@@ -1,24 +1,14 @@
 .. title:: C-PAC 1.7 → 1.8 pipeline configuration mappings
 
-The following YAML is formatted such that the keys are the pipeline configuration keys in C-PAC ≤ 1.7 and the values are a list of successively deeper keys in a C-PAC 1.8 pipeline configuration. For example,
-    .. code-block:: YAML
-
-       pipelineName:
-         - pipeline_setup
-         - pipeline_name
-    
-    indicates the change, from 1.7
-
-    .. code-block:: YAML
+The C-PAC pipeline configuration was changed to a nested format with import capabilities in v1.8.0.
 
-       pipelineName: pipline-name_value
-    
-    to 1.8
+If you provide C-PAC v1.8 with a pipeline configuration file in the v1.7 format, C-PAC will attempt to update the pipeline for you, creating a directory called ``updated_config`` within your output directory, and including the following files:
+   
+   * the updated pipeline configuration file, with the same name as the original pipeline configuration file
+   * the original pipeline configuration file, suffixed with a timestamp and ``.bak``
+   * any keys from the original pipeline configuration file that were not updated, suffixed with ``.rem``
 
-    .. code-block:: YAML
-
-       pipeline_setup:
-          pipeline_name: pipeline-name_value
+Please be sure to check that the updated pipeline configuration file configures the pipeline as intended.
 
 The following keys are deprecated:
 
@@ -41,6 +31,26 @@ See :ref:`Regressors specification <nuisance-no-gui>` to manually update any of
      * ``runMotionSpike``
      * ``spikeThreshold``
 
+The following YAML is formatted such that the keys are the pipeline configuration keys in C-PAC ≤ v1.7 and the values are a list of successively deeper keys in a C-PAC 1.8 pipeline configuration. For example,
+    .. code-block:: YAML
+
+       pipelineName:
+         - pipeline_setup
+         - pipeline_name
+    
+    indicates the change, from 1.7
+
+    .. code-block:: YAML
+
+       pipelineName: pipline-name_value
+    
+    to 1.8
+
+    .. code-block:: YAML
+
+       pipeline_setup:
+          pipeline_name: pipeline-name_value
+
 The full list of mappings is as follows:
 
 .. literalinclude:: /references/1.7-1.8-nesting-mappings.yml
diff --git a/docs/_sources/user/pipelines/default.rst b/docs/_sources/user/pipelines/default.rst
@@ -0,0 +1,2 @@
+.. literalinclude:: /references/default_pipeline.yml
+   :language: YAML
diff --git a/docs/_sources/user/pipelines/design_a_pipeline.rst b/docs/_sources/user/pipelines/design_a_pipeline.rst
@@ -15,7 +15,7 @@ Design A Pipeline
     * ``targetAngleDeg``
     * ``runSymbolicLinks``
 
-    Mappings for all other C-PAC 1.7 keys can be found :doc:`here </user/1.7-1.8-nesting-mappings>`.
+    Mappings for all other C-PAC 1.7 keys can be found :doc:`here </user/pipelines/1.7-1.8-nesting-mappings>`.
 
 C-PAC offers a graphical interface you can use to quickly and easily modify the default pipeline or create your own from scratch:
 
diff --git a/docs/_sources/user/pipelines/pipeline_config.rst b/docs/_sources/user/pipelines/pipeline_config.rst
@@ -47,11 +47,23 @@ Definitions
 
 Using a Text Editor
 -------------------
+If you want to base a pipeline on another pipeline configuration YAML file, you can specify
+
+.. code:: YAML
+
+   FROM: /path/to/pipeline.yml
+
+in your pipeline configuration file. You can use the name of a :doc:`preconfigured pipeline </user/preconfig>` instead of a filepath if you want to base a configuration file on a preconfigured pipeline. If ``FROM`` is not specified, the pipeline will be based on :doc:`the default pipeline <default>`.
+
+C-PAC will include all expected keys from the pipeline file specified in ``FROM`` (or the default pipeline if none is specified). Any keys specified in a pipeline configuration file will take precedence over the same key in the ``FROM`` base configuration, but all omitted keys will retain their values from the ``FROM`` base configuration.
+
 From terminal, you can quickly generate a default pipeline configuration YAML file template in the directory you are in::
 
    cpac utils pipe_config new_template
 
-You can then edit the file as needed, or leave it as the default. If you want to run the analysis from terminal::
+You can then edit the file as needed. For values that you want to leave at the default, you can either leave the key as-is, or you can remove the key, and C-PAC will automatically use value from the default pipeline configuration (or from the pipeline specified in ``FROM``).
+
+If you want to run the analysis from terminal::
 
    cpac run --pipe_config {path to pipeline config} {path to data config}
 
diff --git a/docs/_sources/user/pipelines/preconfig.rst b/docs/_sources/user/pipelines/preconfig.rst
diff --git a/docs/_sources/user/preprocessing.rst b/docs/_sources/user/preprocessing.rst
@@ -6,7 +6,7 @@ Pre-Processing Steps
 
     <div class="flowchart-container"><object data="../_static/flowcharts/pipeline-individual.svg" type="image/svg+xml"></object></div>
 
-The following index provides links to pages containing overviews and configuration instructions for the various pre-processing steps that C-PAC performs.  For more information about how to set up a pipeline configuration, see the :doc:`pipeline configuration page </user/pipeline_config>`.
+The following index provides links to pages containing overviews and configuration instructions for the various pre-processing steps that C-PAC performs.  For more information about how to set up a pipeline configuration, see the :doc:`pipeline configuration page </user/pipelines/pipeline_config>`.
 
 .. toctree::
    :maxdepth: 1
diff --git a/docs/_sources/user/pypeer.rst b/docs/_sources/user/pypeer.rst
@@ -27,7 +27,7 @@ Configuring CPAC to Run PyPEER
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run PyPEER:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run PyPEER:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/quick.rst b/docs/_sources/user/quick.rst
@@ -40,7 +40,7 @@ In addition to the default pipeline, C-PAC comes packaged with a growing library
 
 Detailed information about the selection of pre-configured pipelines are :doc:`available here </user/preconfig>`.
 
-.. include:: design_a_pipeline.rst
+.. include:: pipelines/design_a_pipeline.rst
 
 Acknowledgments
 ---------------
diff --git a/docs/_sources/user/reho.rst b/docs/_sources/user/reho.rst
@@ -34,7 +34,7 @@ Configuring CPAC to Run ReHo
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run anatomical preprocessing:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run anatomical preprocessing:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/sca.rst b/docs/_sources/user/sca.rst
@@ -50,7 +50,7 @@ Before SCA can be calculated, you **must** first extract a seed time series from
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run SCA:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run SCA:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/tse.rst b/docs/_sources/user/tse.rst
@@ -30,7 +30,7 @@ Configuring ROI Time Series Extraction
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run time series extraction:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run time series extraction:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
diff --git a/docs/_sources/user/vmhc.rst b/docs/_sources/user/vmhc.rst
@@ -48,7 +48,7 @@ Configuring CPAC to run VMHC
 Configuration Without the GUI
 """"""""""""""""""""""""""""""
 
-The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipeline_config>` for C-PAC to run anatomical preprocessing:
+The following key/value pairs must be defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>` for C-PAC to run anatomical preprocessing:
 
 .. csv-table::
     :header: "Key","Description","Potential Values"
