Skip to content

Commit 25d7b90

Browse files
oestebanoesteban
committed
enh: add BIDS-Apps section
1 parent 722d8ce commit 25d7b90

File tree

5 files changed

+602
-0
lines changed

5 files changed

+602
-0
lines changed

‎docs/apps/docker.md

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
!!! important "Summary"
2+
3+
Here, we describe how to run *NiPreps* with Docker containers.
4+
To illustrate the process, we will show the execution of *fMRIPrep*, but these guidelines extend to any other end-user *NiPrep*.
5+
6+
## Before you start: install Docker
7+
8+
Probably, the most popular framework to execute containers is Docker.
9+
If you are to run a *NiPrep* on your PC/laptop, this is the **RECOMMENDED** way of execution.
10+
Please make sure you follow the Docker installation instructions.
11+
You can check your Docker Runtime installation running their `hello-world` image:
12+
13+
```Shell
14+
$ docker run --rm hello-world
15+
```
16+
17+
If you have a functional installation, then you should obtain the following output:
18+
19+
```Shell
20+
Hello from Docker!
21+
This message shows that your installation appears to be working correctly.
22+
23+
To generate this message, Docker took the following steps:
24+
1. The Docker client contacted the Docker daemon.
25+
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
26+
(amd64)
27+
3. The Docker daemon created a new container from that image which runs the
28+
executable that produces the output you are currently reading.
29+
4. The Docker daemon streamed that output to the Docker client, which sent it
30+
to your terminal.
31+
32+
To try something more ambitious, you can run an Ubuntu container with:
33+
$ docker run -it ubuntu bash
34+
35+
Share images, automate workflows, and more with a free Docker ID:
36+
https://hub.docker.com/
37+
38+
For more examples and ideas, visit:
39+
https://docs.docker.com/get-started/
40+
```
41+
42+
After checking your Docker Engine is capable of running Docker images, you are ready to pull your first *NiPreps* container image.
43+
44+
## Docker images
45+
46+
For every new version of the particular *NiPrep* app that is released, a corresponding Docker image is generated.
47+
The Docker image *becomes* a container when the execution engine loads the image and adds an extra layer that makes it *runnable*. In order to run *NiPreps* Docker images, the Docker Runtime must be installed.
48+
<!-- (see `installation_docker`{.interpreted-text
49+
role="ref"}).-->
50+
51+
Taking *fMRIPrep* to illustrate the usage, first you might want to make sure of the exact version of the tool to be used:
52+
53+
``` Shell
54+
$ docker pull poldracklab/fmriprep:<latest-version>
55+
```
56+
57+
You can run *NiPreps* interacting directly with the Docker Engine via the `docker run` interface.
58+
59+
60+
## Running a *NiPrep* with a lightweight wrapper
61+
62+
Some *NiPreps* include a lightweight wrapper script for convenience.
63+
That is the case of *fMRIPrep* and its `fmriprep-docker` wrapper.
64+
Before starting, make sure you [have the wrapper installed](https://fmriprep.readthedocs.io/en/latest/installation.html#the-fmriprep-docker-wrapper).
65+
When you run `fmriprep-docker`, it will generate a Docker command line for you, print it out for reporting purposes, and then execute it without further action needed, e.g.:
66+
67+
``` Shell
68+
$ fmriprep-docker /path/to/data/dir /path/to/output/dir participant
69+
RUNNING: docker run --rm -it -v /path/to/data/dir:/data:ro \
70+
-v /path/to_output/dir:/out poldracklab/fmriprep:1.0.0 \
71+
/data /out participant
72+
...
73+
```
74+
75+
`fmriprep-docker` implements [the unified command-line interface of BIDS Apps](framework.md#a-unified-command-line-interface), and automatically translates directories into Docker mount points for you.
76+
77+
We have published a [step-by-step tutorial](http://reproducibility.stanford.edu/fmriprep-tutorial-running-the-docker-image/) illustrating how to run `fmriprep-docker`.
78+
This tutorial also provides valuable troubleshooting insights and advice on what to do after
79+
*fMRIPrep* has run.
80+
81+
## Running a *NiPrep* directly interacting with the Docker Engine
82+
83+
If you need a finer control over the container execution, or you feel comfortable with the Docker Engine, avoiding the extra software layer of the wrapper might be a good decision.
84+
85+
**Accessing filesystems in the host within the container**:
86+
Containers are confined in a sandbox, so they can't access the host in any ways
87+
unless you explicitly prescribe acceptable accesses to the host. The
88+
Docker Engine provides mounting filesystems into the container with the
89+
`-v` argument and the following syntax:
90+
`-v some/path/in/host:/absolute/path/within/container:ro`, where the
91+
trailing `:ro` specifies that the mount is read-only. The mount
92+
permissions modifiers can be omitted, which means the mount will have
93+
read-write permissions. In general, you'll want to at least provide two
94+
mount-points: one set in read-only mode for the input data and one
95+
read/write to store the outputs. Potentially, you'll want to provide
96+
one or two more mount-points: one for the working directory, in case you
97+
need to debug some issue or reuse pre-cached results; and a
98+
[TemplateFlow](https://www.templateflow.org) folder to preempt the
99+
download of your favorite templates in every run.
100+
101+
**Running containers as a user**:
102+
By default, Docker will run the
103+
container as **root**. Some share systems my limit this feature and only
104+
allow running containers as a user. When the container is run as
105+
**root**, files written out to filesystems mounted from the host will
106+
have the user id `1000` by default. In other words, you'll need to be
107+
able to run as root in the host to change permissions or manage these
108+
files. Alternatively, running as a user allows preempting these
109+
permissions issues. It is possible to run as a user with the `-u`
110+
argument. In general, we will want to use the same user ID as the
111+
running user in the host to ensure the ownership of files written during
112+
the container execution. Therefore, you will generally run the container
113+
with `-u $( id -u )`.
114+
115+
You may also invoke `docker` directly:
116+
117+
``` Shell
118+
$ docker run -ti --rm \
119+
-v path/to/data:/data:ro \
120+
-v path/to/output:/out \
121+
poldracklab/fmriprep:<latest-version> \
122+
/data /out/out \
123+
participant
124+
```
125+
126+
For example: :
127+
128+
```
129+
$ docker run -ti --rm \
130+
-v $HOME/ds005:/data:ro \
131+
-v $HOME/ds005/derivatives:/out \
132+
-v $HOME/tmp/ds005-workdir:/work \
133+
poldracklab/fmriprep:<latest-version> \
134+
/data /out/fmriprep-<latest-version> \
135+
participant \
136+
-w /work
137+
```
138+
139+
Once the Docker Engine arguments are written, the remainder of the
140+
command line follows the [usage](https://fmriprep.readthedocs.io/en/latest/usage.html).
141+
In other words, the first section of the command line is all equivalent to the
142+
`fmriprep` executable in a *bare-metal* installation: :
143+
144+
``` Shell
145+
$ docker run -ti --rm \ # These lines
146+
-v $HOME/ds005:/data:ro \ # are equivalent to
147+
-v $HOME/ds005/derivatives:/out \ # a call to the App's
148+
-v $HOME/tmp/ds005-workdir:/work \ # entry-point.
149+
poldracklab/fmriprep:<latest-version> \ #
150+
\
151+
/data /out/fmriprep-<latest-version> \ # These lines correspond
152+
participant \ # to the particular BIDS
153+
-w /work # App arguments.
154+
```

‎docs/apps/framework.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
2+
## What is BIDS?
3+
4+
[The Brain Imaging Data Structure (BIDS)][bids] is a standard for organizing and describing
5+
brain datasets, including MRI.
6+
The common naming convention and folder structure allow researchers to easily reuse BIDS datasets, re-apply analysis protocols, and run standardized automatic data preprocessing pipelines (and particularly, BIDS Apps).
7+
The [BIDS starter-kit](https://github.com/bids-standard/bids-starter-kit) contains a wide collection of educational resources.
8+
Validity of the structure can be assessed with the online [BIDS-Validator](https://bids-standard.github.io/bidsvalidator/).
9+
The tree of a typical, valid (*BIDS-compliant*) dataset is shown below:
10+
11+
```
12+
ds000003/
13+
├─ CHANGES
14+
├─ dataset_description.json
15+
├─ participants.tsv
16+
├─ README
17+
├─ sub-01/
18+
│ ├─ anat/
19+
│ │ ├─ sub-01_inplaneT2.nii.gz
20+
│ │ └─ sub-01_T1w.nii.gz
21+
│ └─ func/
22+
│ ├─ sub-01_task-rhymejudgment_bold.nii.gz
23+
│ └─ sub-01_task-rhymejudgment_events.tsv
24+
├─ sub-02/
25+
├─ sub-03/
26+
```
27+
28+
## What is a BIDS App?
29+
30+
(Taken from the [BIDS Apps paper][bidsapps_paper])
31+
32+
A BIDS App is a container image capturing a neuroimaging pipeline that takes a BIDS-formatted dataset as input.
33+
Since the input is a whole dataset, apps are able to combine multiple modalities, sessions, and/or subjects, but at the same time need to implement ways to query input datasets.
34+
Each BIDS App has the same core set of command-line arguments, making them easy to run and integrate into automated platforms.
35+
BIDS Apps are constructed in a way that does not depend on any software outside of the container image other than the container engine.
36+
37+
BIDS Apps rely upon two technologies for container computing:
38+
39+
1. [Docker] — for building, hosting as well as running containers on local hardware (running Windows, Mac OS X or Linux) or in the cloud.
40+
1. [Singularity] — for running containers on HPCs (high-performance computing).
41+
42+
BIDS Apps are deposited in the [Docker Hub] repository, making them openly accessible. Each app is versioned and all of the historical versions are available to download. By reporting the BIDS App name and version in a manuscript, authors can provide others with the ability to exactly replicate their analysis workflow.
43+
44+
Docker is used for its excellent documentation, maturity, and the Docker Hub service for storage and distribution of the images.
45+
Docker containers are easily run on personal computers and cloud services.
46+
However, the [Docker Runtime] was originally designed to run different components of web services (HTTP servers, databases etc.) using cloud resources.
47+
Docker thus requires root or root-like permissions, as well as modern versions of Linux kernel (to perform user mapping and management of network resources); though this is not a problem in context of renting cloud resources (which are not shared with other users), it makes it difficult or impossible to use in a multi-tenant environment such as an HPC system, which is often the most cost-effective computational resource available to researchers.
48+
49+
Singularity, on the other hand, is a unique container technology designed from the ground up with the encapsulation of binary dependencies and HPC use in mind.
50+
Its main advantage over Docker is that it does not require root access for container execution and thus is safe to use on multi-tenant systems.
51+
In addition, it does not require recent Linux kernel functionalities (such as namespaces, cgroups and capabilities), making it easy to install on legacy systems.
52+
53+
## Analysis levels
54+
BIDS Apps decouple the individual level analysis (processing of independent subjects) from group-level analyses aggregating participants.
55+
For the analysis of individual subjects, Apps need to understand the BIDS structure of the input dataset, so that the required inputs for the designated subject are found.
56+
Apps are designed to easily process derivatives generated by the participant-level or other Apps.
57+
The overall workflow has an entry-point and an end-point responsible of setting-up the map-reduce tasks and the tear-down including organizing the outputs for its archiving, respectively.
58+
Each App may implement multiple map and reduce steps.
59+
60+
![Apps](../assets/journal.pcbi.1005209.g002.png)
61+
62+
## A unified command-line interface
63+
64+
To improve user experience and ability to integrate BIDS Apps into various computational platforms, each App follows a set of core command-line arguments:
65+
66+
``` Shell
67+
$ <entrypoint> <bids_dataset> <output_path> <analysis_level>
68+
```
69+
70+
For instance, to run *fMRIPrep* on a dataset located in `/data/bids_root` and write the outputs to `/data/bids_root/derivatives/`:
71+
72+
``` Shell
73+
$ fmriprep /data/bids_root /data/bids_root/derivatives/ participant
74+
```
75+
76+
In this case, we have selected to run the `participant` level (to process individual subjects).
77+
*fMRIPrep* does not have a `group` level, but other BIDS Apps may have.
78+
For instance, *MRIQC* generates group-level reports with the following command-line:
79+
80+
``` Shell
81+
$ mriqc /data/bids_root /data/bids_root/derivatives/ group
82+
```
83+
84+
[bids]: https://bids.neuroimaging.io/
85+
[bidsapps_paper]: https://doi.org/10.1371/journal.pcbi.1005209
86+
[Singularity]: https://sylabs.io/singularity/
87+
[Docker]: https://docker.com
88+
[Docker Runtime]: https://www.docker.com/products/container-runtime
89+
[Docker Hub]: http://hub.docker.com

0 commit comments

Comments
 (0)