finished workflow tutorial and reworked some of the docs structure

Author: tclose

new-docs/source/_static/css/custom.css

@@ -0,0 +1,4 @@
+div.nbinput .prompt,
+div.nboutput .prompt {
+    display: none;
+}

new-docs/source/conf.py

@@ -144,6 +144,9 @@
     },
 }
 
+html_static_path = ["_static"]
+html_css_files = ["css/custom.css"]
+
 # Add any paths that contain custom themes here, relative to this directory.
 # html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
 

new-docs/source/explanation/conditional-lazy.rst

@@ -1,4 +1,4 @@
-Conditionals and lazy fields
-============================
+Conditional construction
+========================
 
 Work in progress...

new-docs/source/explanation/design-approach.rst

@@ -0,0 +1,75 @@
+
+Design philosophy
+=================
+
+Rationale
+---------
+
+Scientific workflows often require sophisticated analyses that encompass a large collection
+of algorithms.
+The algorithms, that were originally not necessarily designed to work together,
+and were written by different authors.
+Some may be written in Python, while others might require calling external programs.
+It is a common practice to create semi-manual workflows that require the scientists
+to handle the files and interact with partial results from algorithms and external tools.
+This approach is conceptually simple and easy to implement, but the resulting workflow
+is often time consuming, error-prone and difficult to share with others.
+Consistency, reproducibility and scalability demand scientific workflows
+to be organized into fully automated pipelines.
+This was the motivation behind Pydra - a new dataflow engine written in Python.
+
+History
+-------
+
+The Pydra package is a part of the second generation of the Nipype_ ecosystem
+--- an open-source framework that provides a uniform interface to existing neuroimaging
+software and facilitates interaction between different software components.
+The Nipype project was born in the neuroimaging community, and has been helping scientists
+build workflows for a decade, providing a uniform interface to such neuroimaging packages
+as FSL_, ANTs_, AFNI_, FreeSurfer_ and SPM_.
+This flexibility has made it an ideal basis for popular preprocessing tools,
+such as fMRIPrep_ and C-PAC_.
+The second generation of Nipype ecosystem is meant to provide additional flexibility
+and is being developed with reproducibility, ease of use, and scalability in mind.
+Pydra itself is a standalone project and is designed as a general-purpose dataflow engine
+to support any scientific domain.
+
+Goals
+-----
+
+The goal of Pydra is to provide a lightweight dataflow engine for computational graph construction,
+manipulation, and distributed execution, as well as ensuring reproducibility of scientific pipelines.
+In Pydra, a dataflow is represented as a directed acyclic graph, where each node represents a Python
+function, execution of an external tool, or another reusable dataflow.
+The combination of several key features makes Pydra a customizable and powerful dataflow engine:
+
+- Composable dataflows: Any node of a dataflow graph can be another dataflow, allowing for nested
+  dataflows of arbitrary depths and encouraging creating reusable dataflows.
+
+- Flexible semantics for creating nested loops over input sets: Any Task or dataflow can be run
+  over input parameter sets and the outputs can be recombined (similar concept to Map-Reduce_ model,
+  but Pydra extends this to graphs with nested dataflows).
+
+- A content-addressable global cache: Hash values are computed for each graph and each Task.
+  This supports reusing of previously computed and stored dataflows and Tasks.
+
+- Support for Python functions and external (shell) commands: Pydra can decorate and use existing
+  functions in Python libraries alongside external command line tools, allowing easy integration
+  of existing code and software.
+
+- Native container execution support: Any dataflow or Task can be executed in an associated container
+  (via Docker or Singularity) enabling greater consistency for reproducibility.
+
+- Auditing and provenance tracking: Pydra provides a simple JSON-LD-based message passing mechanism
+  to capture the dataflow execution activities as a provenance graph. These messages track inputs
+  and outputs of each task in a dataflow, and the resources consumed by the task.
+
+.. _Nipype: https://nipype.readthedocs.io/en/latest/
+.. _FSL: https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSL
+.. _ANTs: http://stnava.github.io/ANTs/
+.. _AFNI: https://afni.nimh.nih.gov/
+.. _FreeSurfer: https://surfer.nmr.mgh.harvard.edu/
+.. _SPM: https://www.fil.ion.ucl.ac.uk/spm/
+.. _fMRIPrep: https://fmriprep.org/en/stable/
+.. _C-PAC: https://fcp-indi.github.io/docs/latest/index
+.. _Map-Reduce: https://en.wikipedia.org/wiki/MapReduce

new-docs/source/explanation/hashing-caching.rst

@@ -1,4 +1,4 @@
-Hashing and caching
-===================
+Caching
+=======
 
 Work in progress....

new-docs/source/howto/install.ipynb

@@ -0,0 +1,46 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Real-world example"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is an real-world example of a workflow to pre-process T1-weighted MRI images for further analysis\n",
+    "\n",
+    "Work in progress..."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "wf12",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

new-docs/source/howto/real-example.ipynb

@@ -3,86 +3,32 @@
 Pydra
 =====
 
-Pydra is a new lightweight dataflow engine written in Python.
+Pydra is a new lightweight dataflow engine written in Python, which provides a simple way to
+implement scientific workflows that use a mix of shell commands and Python functions.
+
 Pydra is developed as an open-source project in the neuroimaging community,
 but it is designed as a general-purpose dataflow engine to support any scientific domain.
 
-Rationale
----------
-
-Scientific workflows often require sophisticated analyses that encompass a large collection
-of algorithms.
-The algorithms, that were originally not necessarily designed to work together,
-and were written by different authors.
-Some may be written in Python, while others might require calling external programs.
-It is a common practice to create semi-manual workflows that require the scientists
-to handle the files and interact with partial results from algorithms and external tools.
-This approach is conceptually simple and easy to implement, but the resulting workflow
-is often time consuming, error-prone and difficult to share with others.
-Consistency, reproducibility and scalability demand scientific workflows
-to be organized into fully automated pipelines.
-This was the motivation behind Pydra - a new dataflow engine written in Python.
-
-History
--------
-
-The Pydra package is a part of the second generation of the Nipype_ ecosystem
---- an open-source framework that provides a uniform interface to existing neuroimaging
-software and facilitates interaction between different software components.
-The Nipype project was born in the neuroimaging community, and has been helping scientists
-build workflows for a decade, providing a uniform interface to such neuroimaging packages
-as FSL_, ANTs_, AFNI_, FreeSurfer_ and SPM_.
-This flexibility has made it an ideal basis for popular preprocessing tools,
-such as fMRIPrep_ and C-PAC_.
-The second generation of Nipype ecosystem is meant to provide additional flexibility
-and is being developed with reproducibility, ease of use, and scalability in mind.
-Pydra itself is a standalone project and is designed as a general-purpose dataflow engine
-to support any scientific domain.
-
-Design goals
-------------
-
-The goal of Pydra is to provide a lightweight dataflow engine for computational graph construction,
-manipulation, and distributed execution, as well as ensuring reproducibility of scientific pipelines.
-In Pydra, a dataflow is represented as a directed acyclic graph, where each node represents a Python
-function, execution of an external tool, or another reusable dataflow.
-The combination of several key features makes Pydra a customizable and powerful dataflow engine:
-
-- Composable dataflows: Any node of a dataflow graph can be another dataflow, allowing for nested
-  dataflows of arbitrary depths and encouraging creating reusable dataflows.
+See the :ref:`Design philosophy` for more an explanation of the design
+philosophy and goals of Pydra.
 
-- Flexible semantics for creating nested loops over input sets: Any Task or dataflow can be run
-  over input parameter sets and the outputs can be recombined (similar concept to Map-Reduce_ model,
-  but Pydra extends this to graphs with nested dataflows).
-
-- A content-addressable global cache: Hash values are computed for each graph and each Task.
-  This supports reusing of previously computed and stored dataflows and Tasks.
-
-- Support for Python functions and external (shell) commands: Pydra can decorate and use existing
-  functions in Python libraries alongside external command line tools, allowing easy integration
-  of existing code and software.
+Installation
+------------
 
-- Native container execution support: Any dataflow or Task can be executed in an associated container
-  (via Docker or Singularity) enabling greater consistency for reproducibility.
+Pydra itself is a pure-Python package, which has only a handful of dependencies,
+therefore, it is straightforward to install via pip
 
-- Auditing and provenance tracking: Pydra provides a simple JSON-LD-based message passing mechanism
-  to capture the dataflow execution activities as a provenance graph. These messages track inputs
-  and outputs of each task in a dataflow, and the resources consumed by the task.
+.. code-block:: bash
 
-.. _Nipype: https://nipype.readthedocs.io/en/latest/
-.. _FSL: https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSL
-.. _ANTs: http://stnava.github.io/ANTs/
-.. _AFNI: https://afni.nimh.nih.gov/
-.. _FreeSurfer: https://surfer.nmr.mgh.harvard.edu/
-.. _SPM: https://www.fil.ion.ucl.ac.uk/spm/
-.. _fMRIPrep: https://fmriprep.org/en/stable/
-.. _C-PAC: https://fcp-indi.github.io/docs/latest/index
-.. _Map-Reduce: https://en.wikipedia.org/wiki/MapReduce
+    $ pip install pydra
 
+Of course, if you use Pydra to execute shell-commands tools, you will need to either have
+those commands installed on the execution machine, or use software containers
+(e.g., Docker or Singularity) to run them.
 
 
 Indices and tables
-==================
+------------------
 
 * :ref:`genindex`
 * :ref:`modindex`
@@ -94,7 +40,7 @@ Indices and tables
     :hidden:
 
     tutorial/execution
-    tutorial/task
+    tutorial/python
     tutorial/shell
     tutorial/workflow
 
@@ -103,7 +49,7 @@ Indices and tables
    :caption: How-to Guides
    :hidden:
 
-   howto/install
+   howto/real-example
    howto/create-task-package
    howto/port-from-nipype
 
@@ -112,6 +58,7 @@ Indices and tables
    :caption: Explanation
    :hidden:
 
+   explanation/design-approach
    explanation/splitting-combining
    explanation/typing
    explanation/hashing-caching

new-docs/source/index.rst

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Executing tasks\n",
+    "# Running tasks\n",
     "\n",
     "A *Task* is the basic runnable component in Pydra, and can execute either a Python function,\n",
     "shell command or workflows consisting of combinations of all three types."

new-docs/source/tutorial/execution.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Task design"
+    "# Python-task design"
    ]
   },
   {